Randomized controlled trials are kind of a big deal in development economics right now. A recent article in The Economist shows a sizeable rise in the use of RCTs in economics overall over the last 15 years, and recent analysis by David McKenzie shows that RCTs make up a large minority of development papers in top journals (see the figures below).
Source: The Economist on the left; McKenzie (2016) on the right.
In his new book Experimental Conversations: Perspectives on Randomized Trials in Development Economics, Tim Ogden has assembled interviews with a distinguished group that interacts with RCTs in every imaginable way: you have those who pioneered the use of the method in development economics, the next generation of researchers, the chief critics of the method, and consumers of development RCTs at organizations like GiveWell, the Ford and Grameen Foundations, and the Center for Global Development. You also hear from one broader observer of economics as a field (Tyler Cowen) and one of the scholars who pioneered the use of RCTs in U.S. policy (Judy Gueron), to give added perspective.
But although RCTs are at the center of the book, they more often form the starting point for much deeper questions of “how we learn about the world, what evidence is and means, and how policy should and shouldn’t be formed.” Interviewees discuss a wide range of topics, such as the interaction between theory, prior beliefs, and what we test in the field, the role of descriptive data in changing policy, the value of fieldwork (independent of randomization), and the role of RCTs in promoting innovation (see excerpts below).
The interviews have common threads. Ogden repeatedly asks the experts how they believe research affects policy (their “theory of change”) and whether RCTs are more or less effective at doing that than other evidence. The short answer is that views vary, but most endorse a more insightful, nuanced view of policy impact. Here are two views. First, from Esther Duflo: “When, say, a government is interested in the result of my experiment, my experience is that they are not usually interested in taking it wholesale and applying it; they are interested in doing their own experiment. There is this sort of, ‘experiment begets experiment.’” And second, from Nancy Birdsall: “My crude theory about the impact of research is not that research feeds into policy directly. Research kind of goes up in into the air, into the clouds. If it’s good, if it’s interesting, and perhaps especially if there is some compelling behavioral story behind it, as is often the case with an RCT, it gets circulated and cited. Eventually it filters down to World Bank operational people and donor staff and people in the ministries.”
Peppered throughout are recommendations for other books, both those authored by the interviewees – including Judy Gueron’s Fighting for Reliable Evidence, on the history of RCTs and public policy in the U.S. (which gets a recommendation from Angus Deaton: “I learned a lot from that”) and Rachel Glennerster’s Running Randomized Evaluations, plus several others – and those authored by others but relevant to the conversation, such as Charles Kenny’s Getting Better.
One feature/bug of the fact that Ogden is director of the Financial Access Initiative is that his interests center on, well, financial access (microcredit, savings, insurance). This drives both the selection of interviewees – particularly among the researchers – and the course of the conversation. In fairness, the simultaneous publication of 6 microcredit RCTs was an important moment in the history of RCTs in development. But the cast of characters would have been somewhat different from someone whose principal interest is education or health. Education and health both receive mention – in interviews with Banerjee, Duflo, Dupas, Glennerster, Kremer, and others – but they aren’t the focus. One other minor critique is that several of the interviews feel slightly dated. Chris Blattman says, “I’ve only actually published one experiment,” whereas I count 6 published plus 2 working papers on Blattman’s site. But don’t let that dissuade you. You don’t read this book to become familiar with the last 2 years of evidence on savings interventions; you read it to understand fundamental work on savings interventions and – more centrally – how we can think about learning from interventions and translating learning into policy. Indeed, my favorite part of each interview was the discussion of the big questions more than the specific research, but the specific research shows where the big ideas are grounded.
Ultimately, this collection provides great insight into many of the great minds of those producing and using development economics in our time. I highly recommend it. And once in a while, the filter of the economist’s mind amuses, as in Pascaline Dupas’s reformulation of the golden rule: “Maybe not helping a friend in need is extremely costly to people.”
On a personal note, I derived great pleasure from reading the book. Michael Kremer was my advisor in graduate school, Esther Duflo was a mentor, and they along with Abhijit Banerjee all taught me development economics. Chris Blattman, Pascaline Dupas, Jon Robinson, and Dean Yang are all friends of mine dating back to graduate school days. Rachel Glennerster and I, despite never having formally worked together, have participated together in a wide range of events, and I consistently draw on her work and advice. Lant Pritchett advised me when I was trying to decide on my first job. Nancy Birdsall and Jonathan Morduch interviewed me when I was on the academic job market. I work with David McKenzie and Xavi Gine at the World Bank. I read and was inspired by Alex Counts’s book Give Us Credit as an undergraduate. And once I had the daunting task of being a discussant for one of Angus Deaton’s papers, when I was still a graduate student. He was very kind. I’ve had the pleasure of watching much of the work documented here unfold in real time, and this book captures the evolution well.
A few excerpts
- The interaction between theory, prior beliefs, and what we test in the field: “The optimal design of a set of experiments depends very much on how strong your priors are…. Depending on how tight your priors are, your experimental design could be either mostly driven by your theory or mostly driven by, ‘I have no idea, let me try a bunch of different things.’” (Abhijit Banerjee)
- The role of descriptive data in changing policy: “Even when the information is there, it takes a long time to get people to change their priors on what is the problem they face.” (Nancy Birdsall)
- “Often just very good descriptive data that focuses people’s attention on something they haven’t focused attention on before has changed people’s minds in policy as much as any experiment.” (David McKenzie)
- The value of fieldwork: “But what I really like about running field experiments is that…if you want to do a good job, you have to get involved in the details. Which means that you get constant feedback on what is feasible and what is implementable or what is practical and what is just a pipe dream.” (Antoinette Schoar)
- “In the policy world there’s a lot of respect for actually having gone and collected the data and seeing what is happening. I’ve talked to lots of bureaucrats. When they realize you know a fair amount about something, you get a lot of respect. Some of that comes from understanding you did an RCT, some of it comes from the fact that you’ve been and seen a place that they’ve never been.” (Abhijit Banerjee)
- RCTs and innovation: “The modern movement for RCTs in development economics…is about innovation, as well as evaluation. It’s a dynamic process of learning about a context through painstaking on-the-ground work, trying out different approaches, collecting good data with good causal identification, finding out that results do not fit pre-conceived theoretical ideas, working on a better theoretical understanding that fits the facts on the ground, and developing new ideas and approaches based on theory and then testing the new approaches.” (Michael Kremer)
- I’ve excerpted a few passages that struck me in the book, on the respective value of theory versus field experiments, why people don’t take up savings products, 3 questions to guide every field experiment, whether the war over RCTs has been won, and how RCTs lead to innovation.
- You can read the full interview with Angus Deaton here.
- You can read excerpts from the interview with David McKenzie here and here.
- Tyler Cowen: “Truly excellent book…. Interviews are so often so much better than just letting everyone be a blowhard, and Ogden did a great job.”
Join the Conversation