This paper conducts a panel analysis to investigate the relationship between farm size and productivity over a relatively wide range of farms (0-40ha) in Nigeria. We account for observed and unobserved heterogeneities suspected for commonly observed variables in the literature. We further account for the dynamic movement between small-scale and medium-scale farm operations and their productivity over time. Our findings show there is little difference in mean productivity across various farm size categories up to at least 40 hectares. However, there is substantial heterogeneity in within-group productivity. We also find a non-trivial percentage of farms grow or shrink in size with little observed change in productivity over time after accounting for exogenous factors including weather. We conclude that productivity does not depend on farm size, rather, non-size related factors are much more important drivers of productivity. The potential agricultural and developmental policy implications are explored.