Although the inverse farm size-productivity relationship (IR) is sometimes used to motivate arguments in favor of smallholder-led agricultural development, it remains unclear what drives this relationship. It may be attributed to market imperfections that compel small farms to use land more intensively than large farms. Using a three-wave longitudinal household survey from Tanzania, we examine whether the intensity of the IR is related to local factor market activity for land, labor, credit, and animal and machine traction. The IR is evident in Tanzania, although it disappears when family labor is valued at the prevailing local agricultural wage rate. This suggests that labor market imperfections (possibly linked to other market failures) drive the IR. Furthermore, the IR is significantly weakened in the presence of relatively active markets for most factors of production. This suggests that the IR is at least partly driven by imperfections in rural factor markets, underscoring the importance of strategies to improve the functioning of these markets.