This research aims to assess the Total Factor Productivity (TFP) of the whole agricultural sector in Egypt for the period 1961-2012 using Törnqvist index calculations. Particularly, it aims to investigate: (1) the most important factors explaining the TFP growth in the Egyptian agriculture (2) estimating changes in technical efficiency and technical change and determining the magnitude of their contribution to the overall TFP growth, and lastly, (4) based on these findings, providing policy implication and recommendations that allows enhancing and sustaining future growth of agricultural production in Egypt. The currently analysis provided relevant results which might help us understanding the structural trend of the Egyptian agricultural sector, and understanding the most significant variables affecting this trend. Such results will have important policy implications for promoting further growth in the Egyptian agricultural sector. The empirical findings showed that rural development variables were found to significantly and negatively affect agricultural productivity. This demonstrates that agricultural activity is still a marginalized activity which is linked to low levels of income and is a source of employment for low productive labor. Moreover, a negative significant effect of the infrastructure variable on the productivity gains of the agricultural sector in Egypt was found which might indicates a form of low integration of farmers within large neighboring markets. These findings highlighted the decisions makers to take a deeper look at their rural infrastructure strategy, knowing that it may affect the productivity of the agricultural sector as whole.