Climate change continues to pose a threat to food and nutrition security for many households in sub-Saharan African (SSA) countries. Several studies have therefore examined the challenges of climate change to agricultural productivity and poverty, and the impacts of adaptation to climate change on outcomes such as farm yields, net farm incomes, as well as food and nutrition security of households in SSA. In this paper, I review this voluminous literature on the challenges and adaption to climate change by farmers in the region. I will first provide an overview on the literature on the challenges and adaptation to climate change within the last two decades, followed by a brief discussion of the methods used in modeling the impact of adaptation to climate change on farm performance. Finally, I will present some empirical results on the impact of adoption of climate-smart techniques on crop yields by farmers in Ghana, and provide some thoughts on policy implications and future research in the area.