Randomized control trials (RCTs) have been grabbing a lot of headlines lately. Esther Duflo, the principal champion of RCTs and recipient of the John Bates Clark medal, recently had a profile in The New Yorker (gated) and also gave an entertaining TED Talk. As seems to usually be the case with innovations in economics, the discourse has quickly gotten pugilistic.
One of the first punches thrown was by Angus Deaton, a professor of development economics at Princeton. Back in 2008 he gave a presentation with a slide showing Esther Duflo and one of her colleagues jumping out of an airplane, Duflo with a parachute and her colleague without. Deaton was making the point that we don't really need an RCT to know that a parachute is necessary. Duflo has proven to be equally good at landing a punch or two. In her TED Talk, she compared development economics before the advent of RCTs to medieval doctors using leeches. And in her New Yorker profile, she quickly dismisses an argument made in favor of microfinance that sees the fact that the poor are eager to take out microcredit as a positive sign. This is, according to Duflo, "the moronic revealed-preference argument."
Leaving aside for a second the arguments for and against RCTs, I want to ask why it is this particular moment in time that RCTs have managed to get so much attention. It is not as though they are entirely new -- RCTs have been used for around a decade to assess a number of development interventions, e.g. deworming, bednets, female electoral quotas, etc. My guess is that it has a lot to do with the recent and widely publicized RCTs of microfinance, in particular an evaluation of an Indian microfinance outfit last year that found little positive impact on the recipients of microcredit.
The difference between the microfinance RCT and all the others? The RCTs of deworming, etc. all found that these interventions were successful and deserving of continued government and/or aid support. It was only in the case of microfinance that the RCT found little evidence of development impact -- and this prompted a minor public relations scandal, with a number of prominent MFIs issuing a less-than-satisfactory rebuttal.
Microfinance will surely not be the last "victim" of RCTs. Duflo and friends get a lot more publicity from debunking interventions than from showing what works (though this is of course a function of the press and not Duflo et al.). It is only a matter of time before a new RCT finds another development intervention that doesn't match the hype. I expect the next "victim" will have the following characteristics:
- Highly optimistic -- and verifiable -- claims will have been made on behalf of the intervention (see the claims Duflo et al. target in their paper on the microfinance RCT).
- The intervention will have a strong theoretical appeal (e.g. the appeal of microfinance in the way that it is supposed to handle problems of adverse selection through group lending.)
- It will have institutionalized supporters that see the results of the RCT as threatening to their work.
If I had to wager a guess, I would bet on the One Laptop Per Child program as the next "victim". (I await eagerly the results of an ongoing evaluation at the IADB). But there must be many evaluations going on out there I am not aware of. Any suggestions from readers?
Update: In addition to the limitations pointed out by Eric and Helen in the comments and the post I point to by Bill Easterly, there is another methodological problem with RCTs that just occurred to me. At least in the cases I've seen, the selection of partner organization (Spandana, in the case of the Indian microfinance RCT) itself is not random. And this introduces selection bias into the process.
The New Yorker article discusses how Duflo et al. had to work to find willing partner organizations ("...In 2005, after a lengthy search in an industry wary of subjecting itself to this kind of scrutiny...") It seems highly unlikely that those MFIs that have agreed to undergo an RCT are representative of all MFIs. This would seem to severely limit our ability to generalize from the case of Spandana to all MFIs. I suspect someone else has probably already made this point, but I haven't seen it anywhere before.
Join the Conversation