Artificial intelligence and robotics (AI/R) have the potential to result in great change of livelihoods. While individual impacts of AI/R on, for instance, employment, have been subject to a lot of research, how AI/R may affect the poor is scarce. This paper aims to draw attention to how AI/R may impact the poor and marginalized and highlights research needs. A thought experiment compares the future situation of the poor in an AI/R scenario to a scenario without AI/R. A framework is established that depicts poverty and marginality conditions of health, education, public services, work, small businesses including farming as well as voice and empowerment of the poor. This conceptual framework identifies points of entry of AI/R, and is complemented by a more detailed discussion of the way in which changes through AI/R in these areas may relate positively or adversely to the livelihood of the poor. This paper concludes that empirical scenarios and modelling analyses are needed to better understand the different components in the emerging technological and institutional AI/R innovations and identify how they will shape the livelihoods of poor households and communities.