Loss(t) in translation? Measuring disaster crop losses in surveys

Each year, crop production losses due to disasters cause economic losses in the billions of dollars and a loss in kilocalories that could potentially cut undernourishment in low-income countries (LICs) in half. By any definition, these losses are as significant from an aggregate economic perspective as they are a threat to the livelihoods of the poorest. Climate change threatens to increase this burden only further. A fine-grained and standardized way to measure crop losses is essential to inform official statistics and policy and ultimately combat food insecurity among smallholder farmers . 

The Food and Agriculture Organization (FAO) has proposed a new standardized methodology for measuring disaster losses using agricultural surveys. We put this FAO proposal to the test in a recent working paper with data on almost 20,000 plots in Ethiopia, Malawi, and Uganda between 2010 and 2020 from the Living Standards Measurement Study – Integrated Surveys on Agriculture (LSMS-ISA).  

What do we learn about crop losses from survey data? 

Taking the survey data at face value, we find that losses matter a lot: In normal years, maize production could have on average been up to a third higher if it wasn’t for losses . Losses substantially increase in the face of a severe drought recorded in Ethiopia in 2015, with our estimates suggesting that production could have been over 60% higher without the severe drought.  

However, estimating crop production losses is a complex task. For example, farmers face a cognitively challenging problem when asked to estimate the share of harvest they lost. Differences in farming practices between farmers or even plots on the same farm, microclimates and subjective perceptions create variation in (self-reported) shock exposure and impact. 

Are crop losses measured accurately?  

We encounter two puzzling patterns in the data. First, there is a disconnect between farmers’ reports of losses and of realized yields. Intuition would suggest that the attainable harvest, the counterfactual crop production in the absence of any adverse disaster impact, should not change much between years. Therefore, higher losses in disaster years should be reflected in lower realized yields in those years. However, this is hardly the case (Figure 1). For example, farmers’ declared losses in 2015 are consistent with the severe drought in Ethiopia. Yet, this is not reflected in realized yields which are no lower than in other years. Second, while farmers seem to report more losses in disaster years, their reports only very imperfectly match with the assessments of enumerators trained to estimate crop losses. Without a reliable “gold standard” measurement against which to benchmark different estimates, we cannot assess the accuracy of either.   

_{Figure 1: Farmers' reported average realized production and losses in Ethiopia. The total height of the bar represents our estimate of the attainable harvest, the realized harvest (in green) plus any losses due to crop damage (red) and losses in planted area (yellow). The chart excludes plots where the harvest was entirely lost.} 

Is the occurrence of disasters measured accurately? 

The second step to measuring disaster-related crop losses in survey data is the attribution of losses to specific disasters or shocks. Here, we note that different measures of shock exposure based on community reports, a household-level question on shock occurrence, satellite-based rainfall data and farmer reports show only limited agreement. This inconsistency may in part reflect different definitions of specific disasters or shocks, depending on the question asked. Furthermore, farmers that live in the same village oftentimes do not agree whether their plots were exposed to drought with observable plot characteristics only partially explaining differences in reports. This highlights the need for a common definition or threshold for when losses should be classified as attributable to a specific disaster. 

What do we take away? 

What may explain these puzzles? Farmers’ perceptions of disasters and their impacts on crop production are necessarily subjective – conditions that one farmer perceives as a drought a different farmer may perceive as typical and expected. Reported losses, in turn, likely depend on the perceived severity of a disaster. Misperceptions and cognitive biases in farmer reporting have been extensively documented in the literature (e.g. here, here, and here). In particular, recall decay, whereby respondents struggle to accurately remember an event that occurred in the past, can lead to measurement error in agricultural surveys.  

But conscientious survey design and innovative measurement tools can help minimize measurement errors. As a first step, we propose redesigning crop losses questions in a way that is easy to understand and reduces cognitive burden. We also see a lot of promise in moving towards higher-frequency surveys and more responsive monitoring systems, integrated with periodical agricultural and household surveys. Phone surveys are one way to increase the frequency of data collection at an affordable cost.  

Human reporting can further be supported by tools and technology to create more objective benchmarks for monitoring the occurrence of disasters and their impacts on crop production. In situ weather sensors at the community level would allow us to monitor weather-related shocks objectively and in real time. Crop cutting protocols may be amended to capture production losses. Finally, remote sensing used alongside surveys for ground truthing can offer new ways to estimate disaster-induced crop production losses.  

The benefits of a system integrating regular face-to-face surveys with higher frequency data collection over the phone and in situ and remote sensors would extend beyond agriculture , addressing a wider range of concerns such as climate change, health, and food security.