Farmers face many risks in economic decision-making. Therefore, understanding farmers’ risk attitudes is important to support decision-making and policy. Economic experiments have become popular to elicit farmers’ risk preferences. However, previous research is inconclusive about the power of simple lotteries or survey questions to predict actual behavior of farmers. In this paper, we experimentally compare the predictive power of four different lottery tasks. In a 2 x 2 full factorial experimental design, we compare the effect of framing the task in an agricultural context vs. an abstract task, as well as the effect of incentivizing the lottery vs. not using monetary incentives. We also introduce three survey items that ask respondents to rank their risk attitude in different domains. We compare these measures against a benchmark of actual risk management instruments farmers are using. An incentivized lottery without contextual framing triggers most risk-seeking behavior among farmers. However, all four lotteries and three survey-based measures correlate only poorly with the use of actual risk management instruments such as hail insurance. Our findings cast doubt on the predictive power of commonly used risk elicitation instruments. Additional methods are necessary to establish greater external validity in the elicitation of farmers’ risk attitudes.