Have Descriptive Development Papers Been Crowded Out by Impact Evaluations?

|

This page in:

During our August break, there was an interesting discussion on twitter after Scott Cunningham tweeted that “Seems like the focus on identification has crowded out descriptive studies, and sometimes forced what would be otherwise a good descriptive study into being a bad causal study. It's actually probably harder to write a good descriptive study these days. Stronger persuasion req.”

Others quickly pointed to the work by Piketty and Saez, and by Raj Chetty and co-authors that have used large administrative datasets in developed countries to document new facts. A few months earlier, Cyrus Samii set up a thread on descriptive quantitative papers in political science.

But the question got me thinking about recent examples of descriptive papers in development – and the question of what it takes for such papers to get published in general interest journals. Here are some examples published over the last ten years, including some very recently:

Strategy 1: Macro-development - describe key findings at a global level from new databases arising from comprehensive cross-country data collection efforts. Examples include:

Strategy 2: Generalized Micro-development- Pull together/collect a variety of microeconomic datasets from different countries in order to provide new stylized facts. This could include putting together datasets from a range of different impact evaluations, or LSMS datasets, etc. Examples include:
  • Banerjee and Duflo’s papers on the Economic Lives of the Poor (2007) and the Middle Class around the World (2008), both published in the Journal of Economic Perspectives. In both cases they pull together micro studies from a range of countries, and then document patterns about how people in developing countries live their lives.
  •  A similar strategy is used by Bold et al. (2017), also in the Journal of Economic Perspectives, who use data collected in primary schools in seven sub-Saharan African countries to describe what teachers know and do in the classroom.
  • Lagakos et al. on the pattern of lifecycle wage growth across countries, published in the Journal of Political Economy in 2018 – they use household surveys from 18 countries (which is why I didn’t put this under strategy 1, although it has a macro flavor) to document that experience-wage profiles are on average twice as steep in rich countries as in poor countries.
  • Bloom et al.’s  paper on management in schools, published in the Economic Journal in 2015. Here they document the association between better management and education outcomes in surveys they collected in 1,800 schools in 8 countries, including Brazil and India.
  • I just had a paper on firm death in developing countries (joint with Anna Luisa Paffhausen) accepted at the Review of Economics and Statistics. Here we put together 16 panel surveys from 12 developing countries to establish patterns on firm death. This used data from both large country panels like the IFLS, as well as from a number of my impact evaluations.
  • I used the same strategy of pulling together data from a number of different impact evaluation datasets to write a descriptive paper on business practices in small firms in developing countries (with Chris Woodruff), that we published in Management Science (2017).
One key thing here is to help address the generalizability concern readers often have about descriptive findings from just one country by trying to put together the micro-data from a number of different countries. A key issue is then being able to get comparable questions measured across a range of different countries. I added common sets of questions to surveys I was doing for impact evaluations in multiple countries to enable this – for example, adding a small module on why firms close down, and measuring business practices using the same set of questions across countries. Note also that several of these papers include a mix of rich and poor countries, and especially include the U.S. for comparison – indeed in our ReStat paper we were asked to compare our findings to what U.S. evidence has, and in our business practices paper, were asked what we thought these practices would look like in the U.S.

Strategy 3: Focus in detail on a single country, preferably a big one that lots of people care about, or one with data that is amazing.
There are lots of examples of descriptive papers using the amazing linked panel datasets from Scandinavian countries, or that focus on describing economic patterns in countries like the U.S. There are far fewer examples I can think of for developing countries. Work describing key aspects of the development process in India and China seems the most likely to attract general attention here. There are of course many more examples in field journals such as Chen and Ravallion’s 2007 JDE paper exploring progress in lowering poverty in China over several decades; Deaton and Dreze’s 2009 Economic and Political Weekly paper on food and nutrition in India; and Galasso et al.’s forthcoming WBER paper on the dynamics of child development in Madagascar. Dave Evans and Abhijeet Singh offer several other examples.

Final reflections – what does it take for these papers to be general interest?
There are many examples of descriptive work on the U.S. that delves into racial gaps in educational achievement and income; mortality differences by income; wage trends and income inequality over time; intergenerational mobility; etc. The premium for research on the U.S. in top general interest journals means that it is difficult to see the same questions attracting general interest if done for most developing countries (or indeed for most other developed countries). So you need to either be looking at economic questions that can be best answered in developing country settings; drawing an explicit contrast with what we think we know about the world just from rich countries; or else collecting innovative data that documents economic behavior in a way that hasn’t previously been possible. The last is something for readers working on impact evaluations to think about – given all the effort devoted to data collection in many impact evaluations, there is likely scope for more useful descriptive pieces to also come out of these efforts – as suggested by Dave Evans’s blog on getting more out of baselines.

I would also finally note that the bar is even higher for these papers as a job market paper - because they often rely much more on undervalued skills (careful data collection and ability to synthesize data in a clear way) and less on demonstrating you can use a lot of different fancy methods (although of course you can add use structural modelling, machine learning, cutting edge econometric techniques, and some applied theory as ways of describing the data - it is just that you often don't need to).
 

Authors

David McKenzie

Lead Economist, Development Research Group, World Bank

Lant Pritchett
September 05, 2018

A very nice piece in many ways. However, it makes no effort to address the question in the title. It seems you answer is: "Yes, descriptive studies have been crowded out big time by impact evaluations, here are three strategies for overcoming that now massive bias." Of course pointing to various descriptive studies that have been studies just means they have not been completely and totally crowded out, they still may have been massively crowded out. For that one would of course need a counter-factual--what descriptive studies would have been published (in economics journals of quality threshold X) in the absence of the exuberance (irrational?) for RCTs.
So this piece is consistent with the view: "The enthusiasm for RCTs has driven out descriptive pieces in favor of "rigor" on less important topics and that is a bad thing for economics and development generally, but here, for the descriptive paper minded, are three strategies for overcoming that bias."

David McKenzie
September 05, 2018

Fair enough, I totally agree that my post only establishes that they haven't been completely crowded out, and not what the counterfactual might be if RCTs didn't exist. In my comment on Esther's chapter for a book that is way too long forthcoming (https://drive.google.com/file/d/0B9C9RwWKZrUNbUY3Z3JSSUxyRk0/view) I build on her point where she shows that the number of development papers in top journals grew by the number of RCTs, so that the number of non-RCT development papers did not fall as RCTs started. I show that of out of the 454 development papers published in 14 journals in 2015 (top-5 + ReStat/AEJApp/EJ + JDE/EDCC/WBER/WD), only 44 are RCTs (9.7%). The consequence is that RCT-studies are only a small share of all development research taking place. Moreover, as my post notes, the bias towards studies on the U.S. in general interest journals suggests that the bigger factor might be that they crowd out all development work (RCT or not). That said, it is hard to know what the right counterfactual is to compare to.

Lant Pritchett
September 05, 2018

You intimate that there is a "generalizability" concern about descriptive studies which limits their publication in general interest journals, but that does not explain why RCTs do get in as the generalizability problems are just as severe. But, just one example the general interest journal AEJ:Applied Economics published a paper in which the "treatment" was establishing "village based" schools in 13 (!?) villages in one province (Ghor) of one country (Afghanistan). And that finding is worthy of publication in a top economics journal because....of its generalizability? How so?

Chris Blattman
September 06, 2018

My takeaways from this post:
- Descriptive studies were never easy to publish outside of review papers and books
- Descriptive studies remain very difficult to publish
- The treatment effect of randomized trials is not evident, and is probably minor
- You could argue that descriptive studies are complementary to randomized trials. Many trials have resulted in new data on levels and trends of phenomena. Arguably Poor Economics by Banerjee and Duflo is a big descriptive analysis based in Ory on trials. I unexpectedly found that a lot of my own studies (dispute resolution in Liberia, sweatshops in Ethiopia) the randomized trial results were less important than the descriptive analysis.

Karthik Muralidharan
September 06, 2018

I agree with Chris that the two types of studies are complements and not substitutes.
While most of my ongoing work is experimental, 3 of my 4 most-cited papers to date are descriptive papers that presented new facts in representative samples (on teacher & health worker absence in developing countries, and on public and private schools in India).
Those facts in turn helped to motivate a research program (for me and others) on experimental evaluations of promising approaches to improving service delivery in developing countries, speaking to the complementary nature of the two approaches.
The objective function of most journal editors places a large weight on maximizing the expected impact of papers they publish (proxied by citations). So good descriptive papers that present new/important facts that motivate or provide context to future work (and will hence get cited) should be able to publish well. I don't see any reason for crowd-out here.