Agricultural technological improvements are crucial to increase on farm production and thereby reduce poverty. However the use of improper identification strategies on the impacts of improved technologies on farmer welfare could potentially pose a threat to good practice agricultural policy making. In this paper, propensity matching strategies and endogenous switching regression were used to test whether an improved fallow, a soil fertility improving technology that passed the requirements for a high impact intervention based on non randomised impact assessment methodologies could still pass this test. Using data from 324 randomly surveyed households in Chongwe district of Zambia, the rigorous econometric methods confirmed the positive impact of improved fallows on household maize yields, maize productivity, per capita maize yield and maize income. Conflicting results were obtained when a broader welfare indicator – per capita crop income, was considered. Whereas the non-randomised and kernel matching methods showed that per capita crop incomes were significantly higher for the adopters than for the non adopters, the causal effect of improved fallows on this variable was non significant when nearest neighbour matching strategy and the more robust endogenous switching regression were used. It was concluded that the technology improves welfare through increased maize and hence increased food security, and through incomes from the maize crop. The maize income derived from improved fallows were however not sufficient enough to drive the general crop income to significantly higher levels. The need to diversify the use of improved fallows on high valued crops was recommended while the importance of using better and more robust methodologies in evaluating impact of interventions was emphasised.