This is co-authored with Eric Jospe, Jeremy Magruder, and Ruoyi Song
Policy evaluation in the long-run
The impacts of policies can take many years to materialize, yet surveys of policy beneficiaries often stop just a few years after a baseline survey. Such short-run data collection can miss important impacts: a recent experimental evaluation found lower child mortality when parents received a deworming intervention as children. To capture such slow-to-emerge impacts, a growing number of experimental and quasi-experimental evaluations now collect follow-up data on household beneficiaries years and decades after implementation was completed.
While long-term follow-up on households and individuals is increasingly common, long-term follow-up data collection on agricultural land is much rarer. Many important policies target enhancing the productivity of agricultural land, ranging from land tenure regularization to, as we study, irrigation infrastructure. Evaluating these questions requires measuring plot-level crop choice and production, typically via up-to-one year recall during household surveys; despite innovations enabling measurement from satellite imagery of changes in plot boundaries, erosion, or even staple crop yields conditional on crop choice (e.g., here or here), household surveys are often necessary for measurement of outcomes such as crop choice, irrigation use, and sales.
Prima facie, long-term follow-up on the agricultural productivity of land seems much easier: the land itself, unlike a household or individual, cannot move. But land does not answer our survey questions! Following up on agricultural land requires surveying cultivators, who may change when plots are rented or sold. Failing to track plots across cultivators can cause conclusion-altering bias, as productive investments in land can meaningfully affect the selection of cultivators. Fortunately, cultivators must typically live close to their land, facilitating follow up.
In previous work, we estimated the effects of large-scale hillside irrigation schemes on agricultural productivity in Rwanda using a plot-level spatial regression discontinuity design, 2-4 years after the schemes were constructed in 2015. By comparing plots on either side of scheme boundaries, we found that plots with access to irrigation were 16pp-18pp more likely to be irrigated and had 8%-10% higher annual agricultural production. We collected data from both the plots’ baseline managers and also their current cultivators, which required multiple surveys when plots had been sold or rented.
Irrigation, 10 years later
Today in 2025, 10 years after the initial construction of the hillside irrigation schemes, there are many reasons to expect impacts may have evolved. Irrigation infrastructure may have degraded, perhaps due to insufficient maintenance, and households may no longer use irrigation. Alternatively, improvements in infrastructure over time and economic growth may have increased demand for the horticultural crops grown using irrigation. And land markets may have slowly reallocated irrigable land towards farmers interested in using irrigation.
To answer these questions, we decided to follow-up with households and plots that we’d previously surveyed 6 years before. As the hillside irrigation schemes did not move anywhere, we planned to use the same spatial regression discontinuity design to estimate the effects of access to irrigation 10 years later.
Making a 10 year agricultural plot-panel happen
Build on previous protocols: To start, building on existing protocols, questionnaires, and a reproducible codebase for high frequency checks and analysis streamlined preparing for follow-up data collection without starting from scratch. The documentation and organization of our previous follow-up survey served as a natural starting point, and enabled focusing on adapting to new needs and to challenges that arise over such a long-time span.
Everything starts from field mobilization: Preparatory work for the follow-up data collection was essential to minimize and understand attrition. Similar to practices in the Kenya Life Panel Survey, we began with a first round of calls to identify baseline plot managers’ current location. Previous survey rounds had identified phone numbers for backup contacts and for relevant informants in surveyed communities (typically village leaders). These were used to confirm availability of baseline plot managers’, and identify the relevant respondent for both the household and the plot when things had changed: more than 5% of our baseline sample had moved outside the irrigation schemes, and more than 10% had a change to the household head. In other cases, attrition emerged when natural disasters such as landslides had destroyed the plot. Identifying the different scenarios we might encounter before data collection facilitated planning the surveys of both baseline managers and current cultivators. In all cases, we found using detailed descriptions of the plots and the identity of the previous manager for follow up was lower cost than using GPS coordinates, and if anything more accurate due to the small plot sizes in our context.
Prepare for long-term follow-up from the baseline: As in previous rounds, recalling a plot from just its description could have been challenging for households with multiple nearby plots. As a result, in previous rounds we drew a map with households of all of their plots, and brought a printed copy of the map to the household to facilitate recall. By starting the agricultural part of the survey with a presentation of the map, and updating the map with the household, we helped respondents confirm their recall of the plots we asked them about 6 years prior.
Keep records on tracking and field mobilization: 10 years later, we found that 23% of plots in our sample were cultivated by someone other than the baseline manager; we attempted to “track” each of these plots and collected production data from its current cultivator. For each such plot, as in previous rounds we asked the baseline manager for contact information for the household they had sold or rented the plot to, or who they no longer rent the plot in from. In some cases, the plot had been sold multiple times, and multiple rounds of follow up were needed. When we reached a dead end, we visited the plot itself with a local informant to track down the current cultivator. Records we had kept on the tracking process from previous survey rounds were helpful to ensure we had contact information from an informed respondent.
Identify opportunities during long-term follow up: Long-run follow-up in agriculture brings an additional challenge, and opportunity: the impacts of agricultural interventions can vary year-to-year, so multiple years of follow-up data collection can improve external validity over time. For selected outcomes for which longer-term recall was feasible, we asked respondents to recall the past 7 years of crops cultivated, irrigation use, crop failures, and land rentals on their plot. After extensive piloting, we identified that first asking respondents for a major event in each year---whether personal such as marriage or birth, or agricultural such as joining a cooperative---facilitated recall of outcomes anchored to the event. Over 90% of respondents reported confidently recalling these decisions on their plot. For these outcomes, longer-term recall enables us to fill a 10-year panel of crop choice and irrigation use to better understand impacts on dynamics and resilience.
How did it all turn out?
Thanks to the dedication of our field team and the preparatory work leading up to it, we were able to hold plot-level attrition below 5% from our previous follow-up survey 6 years prior. Due to the possibility of following up on new cultivators even when the baseline cultivator has moved, this attrition is lower than we would have experienced due to moves outside our study area. These protocols can enable more long-run follow-up data collection on agricultural land, to evaluate the long-term dividends from investments in agricultural productivity.
The data collection described was generously funded by IGC.
Join the Conversation