​Good food and good economics both start with quality ingredients

Do economists and policy analysts pay enough attention to the quality of the data they work with? The focus in these professions seems to be much more on using and developing sophisticated econometric and statistical models, or pretty data visualization software, than on assessing the quality of the data that are fed into those models and tools (let alone working to improve the quality of the data).
 

Isn’t that akin to a chef worrying more about the quality of the pan, and the appearance of the dish, than about the quality of the ingredients? Top chefs use top quality kitchen utensils, but they also demand the best ingredients. Economists seem rightly concerned with the tools, but not too concerned to check whether the food they are cooking is ripe or rotten. 
 
The Living Standard Measurement Study team in the research department of the World Bank has been collecting living standards and poverty data since the 1980’s, and has traditionally been obsessed with data quality. Recently, it has revamped its program of methodological validations to focus specifically on assessing and improving data quality. The program consists of a number of randomized, methodological experiments in which different methods for collecting data on a specific topic are tested, and compared to some benchmark that is generally known to be very accurate, but too complex or costly to be implemented in large-scale national household surveys. A substantial part of the program focuses on variables of interest to agriculture, but we also have a number of ongoing studies measuring soft skills, food consumption, and a range of other topics that should matter for many of the things we are interested in understanding when we ‘talk development’.
 
One particularly exciting experiment that I worked on recently identified and evaluated a viable method to collect recall data from small-scale livestock keepers in developing countries on the amount of milk they gather for human consumption (whether for their households or for sale).
 

Milk is important for many of the world’s poor, both as a source of nutrients and as a source of cash (see as an example these studies of Uganda and Ethiopia, that will be forthcoming in a Special Issue of the Journal of Development Studies later in 2015). At the same time, it is very hard to get an idea of how much milk a household can collect in a year, and therefore to reach even an approximate estimate of the impact of household milk production on income, livelihoods and nutrition. This is because animals are milked over long periods of time, with milk production having pronounced seasonal patterns partly related to the fact that milking potential peaks shortly after giving birth, and declines thereafter. Furthermore, the milking potential of the animal is of little guidance in terms of estimating the milk that is actually gathered, as farmers may not milk more that they can use or sell, may leave some to the suckling young animals, or may otherwise decide how much milk to extract. Livestock specialists are therefore skeptical of economists and statisticians naively asking farmers: “How much milk did you collect from your animals this year?”

Giovanni Federighi, Kalilou Adamou, Pierre Hiernaux and I therefore set out to try to measure the accuracy of survey measures of milk production, the extent to which they can be relied upon, and how can we improve on them. To do that, we set up a simple system for physically monitoring the milk production (or more accurately, off-take) of 300 cattle owners in rural Niger. The herders received plastic containers and were trained to mark their daily milk off-take. Enumerators visited the herders every fortnight to collect the data over a full year. We use the data collected this way as the benchmark against which to assess the data collected through six recall methods, including the ‘status quo’ and five competing, hopefully improved, scalable approaches.
 
The details of the experiment and the results are available in a recently published working paper, but here are some main takeaways. The first is that even though there is a substantial amount of measurement error in the way even the best recall methods we tested perform, some methods do in fact perform fairly accurately, and much more accurately than what we were expecting. In particular, the methods do a reasonable job at estimating the more common central tendency measures (mean and median), as well as the distribution of milk production across sample households. We were also able to identify two methods that clearly outperformed the others (these are the methods in the top right and bottom left corners in the figure below).
 

What I want to highlight here is that we should be aware that data collection methods matter and can make a big difference at the analysis stage. The marginal returns to investing in better data via more methodological research are likely to be at least not inferior to those from investing in other aspects of analytical methods.
 
A single study cannot have the final word on milk data collection, and more research is needed to settle the argument, but the same can be said for many other topics central to the development debate for which we can only count on a handful of methodological studies: consumption, agricultural productivity, labor, income, subjective welfare, and the list goes on. It might be time for more research funding, journal space, and researchers’ attention to go to the ingredients, not just the utensils.