Have Descriptive Development Papers Been Crowded Out by Impact Evaluations?

During our August break, there was an interesting discussion on twitter after Scott Cunningham tweeted that “Seems like the focus on identification has crowded out descriptive studies, and sometimes forced what would be otherwise a good descriptive study into being a bad causal study. It's actually probably harder to write a good descriptive study these days. Stronger persuasion req.”

Others quickly pointed to the work by Piketty and Saez, and by Raj Chetty and co-authors that have used large administrative datasets in developed countries to document new facts. A few months earlier, Cyrus Samii set up a thread on descriptive quantitative papers in political science.

But the question got me thinking about recent examples of descriptive papers in development – and the question of what it takes for such papers to get published in general interest journals. Here are some examples published over the last ten years, including some very recently:

Strategy 1: Macro-development - describe key findings at a global level from new databases arising from comprehensive cross-country data collection efforts. Examples include:

• Chen and Ravallion on poverty around the developing world, published in the Quarterly Journal of Economics in 2010 – using data from almost 700 household surveys around the world.
• Henderson et al. on the worldwide spatial distribution of economic activity and its association with different geographic variables, published in the Quarterly Journal of Economics in 2018 – using nightlight data at the level of 240,000 grid cells.
• Bick et al. on how hours worked vary with income across and within countries, published in the American Economic Review in 2018 – building a new cross-country dataset on hours worked from different household surveys to establish the new fact that adults in low-income countries work 50 percent more hours per week than adults in high-income countries.

Strategy 2: Generalized Micro-development- Pull together/collect a variety of microeconomic datasets from different countries in order to provide new stylized facts. This could include putting together datasets from a range of different impact evaluations, or LSMS datasets, etc. Examples include:

• Banerjee and Duflo’s papers on the Economic Lives of the Poor (2007) and the Middle Class around the World (2008), both published in the Journal of Economic Perspectives. In both cases they pull together micro studies from a range of countries, and then document patterns about how people in developing countries live their lives.
•  A similar strategy is used by Bold et al. (2017), also in the Journal of Economic Perspectives, who use data collected in primary schools in seven sub-Saharan African countries to describe what teachers know and do in the classroom.
• Lagakos et al. on the pattern of lifecycle wage growth across countries, published in the Journal of Political Economy in 2018 – they use household surveys from 18 countries (which is why I didn’t put this under strategy 1, although it has a macro flavor) to document that experience-wage profiles are on average twice as steep in rich countries as in poor countries.
• Bloom et al.’s  paper on management in schools, published in the Economic Journal in 2015. Here they document the association between better management and education outcomes in surveys they collected in 1,800 schools in 8 countries, including Brazil and India.
• I just had a paper on firm death in developing countries (joint with Anna Luisa Paffhausen) accepted at the Review of Economics and Statistics. Here we put together 16 panel surveys from 12 developing countries to establish patterns on firm death. This used data from both large country panels like the IFLS, as well as from a number of my impact evaluations.
• I used the same strategy of pulling together data from a number of different impact evaluation datasets to write a descriptive paper on business practices in small firms in developing countries (with Chris Woodruff), that we published in Management Science (2017).

One key thing here is to help address the generalizability concern readers often have about descriptive findings from just one country by trying to put together the micro-data from a number of different countries. A key issue is then being able to get comparable questions measured across a range of different countries. I added common sets of questions to surveys I was doing for impact evaluations in multiple countries to enable this – for example, adding a small module on why firms close down, and measuring business practices using the same set of questions across countries. Note also that several of these papers include a mix of rich and poor countries, and especially include the U.S. for comparison – indeed in our ReStat paper we were asked to compare our findings to what U.S. evidence has, and in our business practices paper, were asked what we thought these practices would look like in the U.S.

Strategy 3: Focus in detail on a single country, preferably a big one that lots of people care about, or one with data that is amazing.
There are lots of examples of descriptive papers using the amazing linked panel datasets from Scandinavian countries, or that focus on describing economic patterns in countries like the U.S. There are far fewer examples I can think of for developing countries. Work describing key aspects of the development process in India and China seems the most likely to attract general attention here.

• Das et al.’s (2016) AER paper on health care quality in rural India, where they combine it being on a big country (India) with innovative data collection (audit studies).
• Jayachandran and Pande’s (2017) AER paper on why are Indian children so short, where they document how the height disadvantage varies with birth order and geographic differences in son preference.

There are of course many more examples in field journals such as Chen and Ravallion’s 2007 JDE paper exploring progress in lowering poverty in China over several decades; Deaton and Dreze’s 2009 Economic and Political Weekly paper on food and nutrition in India; and Galasso et al.’s forthcoming WBER paper on the dynamics of child development in Madagascar. Dave Evans and Abhijeet Singh offer several other examples.

Final reflections – what does it take for these papers to be general interest?
There are many examples of descriptive work on the U.S. that delves into racial gaps in educational achievement and income; mortality differences by income; wage trends and income inequality over time; intergenerational mobility; etc. The premium for research on the U.S. in top general interest journals means that it is difficult to see the same questions attracting general interest if done for most developing countries (or indeed for most other developed countries). So you need to either be looking at economic questions that can be best answered in developing country settings; drawing an explicit contrast with what we think we know about the world just from rich countries; or else collecting innovative data that documents economic behavior in a way that hasn’t previously been possible. The last is something for readers working on impact evaluations to think about – given all the effort devoted to data collection in many impact evaluations, there is likely scope for more useful descriptive pieces to also come out of these efforts – as suggested by Dave Evans’s blog on getting more out of baselines.

I would also finally note that the bar is even higher for these papers as a job market paper - because they often rely much more on undervalued skills (careful data collection and ability to synthesize data in a clear way) and less on demonstrating you can use a lot of different fancy methods (although of course you can add use structural modelling, machine learning, cutting edge econometric techniques, and some applied theory as ways of describing the data - it is just that you often don't need to).