Syndicate content

David McKenzie's blog

Have RCTs taken over development economics?

David McKenzie's picture

Last week the “State of Economics, State of the World” conference was held at the World Bank. I had the pleasure of discussing (along with Martin Ravallion) Esther Duflo’s talk on “The Influence of Randomized Controlled Trials on Development Economics Research and on Development Policy”. The website should have links to the papers and video stream replay up (if not already, then soon).

The first part of Esther’s talk traced out the growth in RCTs in development economics. She pointed out that in 2000 the top-5 journals published 21 articles in development, of which 0 were RCTs, while in 2015 there were 32, of which 10 were RCTs – so pretty much all the growth in development papers in top journals comes from RCTs. She also showed that the more recently BREAD members had received their PhD, the more likely they were to have done at least one RCT.
In my discussion I expanded on these facts to put them in context, and argue against what I see as a couple of strawman arguments: 1) that top journals only publish RCTs, and that RCTs have taken over development research; and 2) that young researchers have a “randomize or bust” attitude and refuse to do anything but RCTs. I thought I’d summarize what I said on both here.

Weekly links June 10: hardworking poor people, the practicalities of RCTs, what is clever in marketing experiments, marshmallow tests revisited, and more…

David McKenzie's picture
  • From VoxEU – people in developing countries not only earn a lot less than those in developed countries, they also work longer hours and have less leisure – as a result “Adults in poor countries …are even less productive than we thought.”
  • All the Stata cheatsheets from Geocenter in one place
  • In Scientific American: does financial stress literally hurt?
  • In the Washington Post wonkblog, more evidence why the Marshmallow test is probably not measuring what you think it is – “this uncovers a broader problem with how we perceive the actions of people who live very different lives than we do. We brand certain actions and choices as mistakes, when they might simply be developmental adjustments necessary to cope with their environment. For those who don't worry about their next meal, because they never had to, choosing a marshmallow now instead of two marshmallows in a few minutes, all things equal, could only be the result of impulse-driven folly. For those who do have to worry about the next meal, passing up food now for the promise of food later is the misguided move.”

Weekly links June 3: Small begets big, expert predictions, process evaluation, measurement, and more…

David McKenzie's picture

Book Review: Grit – Takeaways for Development Economists and Parents

David McKenzie's picture

Angela Duckworth’s new book Grit: The Power of Passion and Perseverance has been launched with great fanfare, reaching number two on the NY Times Nonfiction bestseller list. She recently gave a very polished and smooth book launch talk to a packed audience at the World Bank, and is working with World Bank colleagues on improving grit in classrooms in Macedonia. Billed as giving “the secret to outstanding achievement” I was interested in reading the book as both a researcher and a parent. I thought I’d continue my book reviews series with some thoughts on the book.

Weekly links May 20: AEA P&P Special Edition

David McKenzie's picture
The latest AEA papers and proceedings has a number of interesting papers:
  • In the Richard T. Ely lecture, John Campbell discusses the challenge of consumer financial regulation – he distinguishes 5 dimensions of financial ignorance many households exhibit: 1) ignorance of even the most basic financial concepts (financial illiteracy); 2) ignorance of contract terms (such as not knowing about the fees build into credit cards or when mortgage interest rates can change); 3) ignorance of financial history – relying too much on own experiences and the recent past; 4) ignorance of self- a lot of financially illiterate people are over-confident about their abilities; and 5) ignorance of incentives, strategy and equilibrium – failure to take account of incentives faced by other parties to transactions.  Given these problems, and the limits of financial education and disclosure requirements to fix them, he discusses what financial regulation is needed: “consumer financial regulation must confront the trade-off between the benefits of intervention to behavioral agents, and the costs to rational agents….the task for economists is to confront this trade-off explicitly”

What’s New in Measuring Subjective Expectations?

David McKenzie's picture

Last week I attended a workshop on Subjective Expectations at the New York Fed. There were 24 new papers on using subjective probabilities and subjective expectations in both developed and developing country settings. I thought I’d summarize some of the things I learned or that I thought most of interest to me or potentially our readers:

Subjective Expectations don’t provide a substitute for impact evaluation
I presented a new paper I have that is based on the large business plan competition I conducted an impact evaluation of in Nigeria.  Three years after applying for the program, I elicited expectations from the treatment group (competition winners) of what their businesses would be like had they not won, and from the control group of what their businesses would have been like had they won. The key question of interest is whether these individuals can form accurate counterfactuals. If they could, this would allow us a way to measure impacts of programs without control groups (just ask the treated for counterfactuals), and to derive individual-level treatment effects. Unfortunately the results show neither the treatment nor control group can form accurate counterfactuals. Both overestimate how important the program was for businesses: the treatment group thinks they would be doing worse off if they had lost than the control group actually is doing, while the control group thinks they would be doing much better than the treatment group is actually doing. In a dynamic environment, where businesses are changing rapidly, it doesn’t seem that subjective expectations can offer a substitute for impact evaluation counterfactuals.

Weekly links May 6: expensive African cities, lotteries for housing, placebo effects, and more…

David McKenzie's picture
  • I liked the recent Planet Money podcast #698 (a long way home) – there is an interesting discussion of why a lottery is held for access to a housing assistance program in Connecticut, and how they ended up with a lottery rather than other systems of allocating resources – and a great quote about the mishmash of anti-poverty programs in the U.S. which, paraphrasing, is basically “it is not like Congress ever sat down and said what is the best use of the money we set aside to fight poverty” but rather how many different programs have come up over time, all with their own rules and constituencies.
  • The latest Journal of Economic Perspectives has a symposium on inequality beyond income (US focused) and a paper on the billion prices project that I linked to a blog post on last week
  • Should policy seek to promote small firms or large ones in Africa? Frances Teal on the CSAE blog: “Policy rhetoric focuses on the problems faced by small firms. Data from Ghana over the period for which we have it suggests that it is large firms that face the problems. Unloved possibly because they are not seen as beautiful they are vital for the output of the sector. Policy, not for the first time in Africa, seems to be focused on completely the wrong problem.”
  • The Los Angeles Review of Books has a longish discussion on placebo effects in reviewing an anthology on placebos, and how they really don’t work as much of the time in medicine as many people think, and how the term might be over-used in social sciences.
  • New in the working paper series: Is living in African cities expensive? Using data from the “2011 round of the International Comparison Program. Readjusting the calculated price levels from national to urban levels, the analysis indicates that African cities are relatively more expensive, despite having lower income levels. The price levels of goods and services consumed by households are up to 31percent higher in Sub-Saharan Africa than in other low- and middle-income countries, relative to their income levels. Food and non-alcoholic beverages are especially expensive, with price levels around 35 percent higher than in other countries.”

An addendum to pre-analysis plans: Pre-specifying when you won’t use data collected

David McKenzie's picture

Researchers put a lot of effort into developing survey questionnaires designed to measure key outcomes of interest for their impact evaluations. But every now and then, despite efforts piloting and fine-tuning surveys, some of the questions end up “not working”.  The result is data that are so noisy and/or missing for so many observations that you may not want to use them in the final analysis. Just as pre-analysis plans have a role in specifying in advance what variables you will use to test which hypotheses, perhaps we also want to specify some rules in advance for when we won’t use the data we’ve collected. This post is a first attempt at doing so.

Weekly links April 29: claiming your failures, what a billion prices tells us, the demand for health products, and more…

David McKenzie's picture

Weekly links April 22: development engineering, reporting context, the downside of good behavior, and more…

David McKenzie's picture
  • The inaugural issue of Development Engineering is now out (all issues are open access!). I’m delighted that my paper on attempting to use RFID to track small firm sales is in this first issue, along with a paper on how to randomize better in sequential randomized trials, a paper that proposes a “system [which] leverages smartphones, cellular based sensors, and cloud storage and computing to lower the entry barrier to impact evaluation”, a paper on biomass stoves, and one on rural electrification. Note also this from the editor’s introduction “we see major benefits from publishing studies that find weak or no impacts. In global development, there should be no silent failures; there is inherent value in learning from interventions that fail to achieve their intended impacts.”

Pages