Syndicate content

Can randomized control trials reduce poverty?

Shanta Devarajan's picture

If you give milk to schoolchildren and they perform well in school, how do you know it’s because of the milk, or because the children were high achievers anyway, or went to better schools? 

By randomly choosing the children who receive the milk, and comparing the outcomes of this “treatment group” with a “control group” (those that didn’t receive milk), we can get a more accurate measure of the program’s impact than if we were to simply compare the children’s performance before and after they drank milk. 

In a recent issue of Boston Review, Rachel Glennerster and Michael Kremer, two of the leading proponents of these randomized control trials (RCTs), survey what we have learned from them, especially those that show that people sometimes deviate from standard economic assumptions (a burgeoning field known as “behavioral economics”).

At Michael’s and Rachel’s request, Jishnu Das, Jeff Hammer and I wrote a comment, the more complete version of which is here.  We make two points:

  • RCTs tell us what will happen if you intervene; they don’t answer the question: “Should you intervene?”  The answer lies in whether there is a market failure or need for redistribution and, in the latter case especially, whether giving milk to schoolchildren is the best way to improve the welfare of poor people.
  • Even if there is a rationale for intervention, RCTs—which are usually implemented by an NGO that manages a well-defined program—tell us very little about what will happen if the program is implemented by real government officials who are facing political pressures and may not be able to (or even want to) ensure that milk is actually delivered to the children.

Randomized control trials have become the most popular technique in development economics.  Let’s make sure they can be used to contribute to development’s goal—to improve the lives of the world’s poor.

 

Comments

As someone who has worked extensively in building the M&E capacity of grassroots organizations in Africa, the latest trend towards using "gold standard" of randomized control trials is especially troubling when one is talking about community initiatives. Imposing such incredibly risk-averse behavior, evaluating every single intervention on people who are in the process of organizing at the local level is most certainly a drain on their time and scarce resources. What so many people on the ground have told me again and again is that abstract metrics don’t help them understand their relationship to improving the well-being of the people they serve. As members of the community, they read trends through what’s happening on the ground, rather than using any theory. During my time in the development sector as a whole, I've seen an increasing desperation to “know” what is inherently beyond logic and induction. It is certainly time to examine our belief that there are technocratic, precise ways of measuring progress in order to make consequential judgments based on these measures. The increasing obsession with abstract metrics and experimental design, stemming from a reductive, managerial approach in development, is quite far from the intimate, difficult, and complex factors at play at the national and grassroots levels. The business sector seems to have a healthier relationship with risk in their for-profit endeavours, perhaps something we may need to explore in the development sector. Yes, let's pursue and obtain useful data, but at a scale at which information can be easily generated, utilized, and acted upon by those we are trying to serve on the ground. "Those who work selfishly for results are miserable." ~Sri Krishna, Bhagavad Gita, The Song of God

Submitted by olugbenga adesanya on
This could be a useful tool for on site economic management. Africa, especially is in great need of development. Perhaps with a little modification for local issues, RCT would help our famished continent to grow steadily in real terms. I salute your resolve to add value to Africa.

Submitted by Anonymous on
The problem with RCT is that the "control group" has to remain as such during the study period, which poses some ethical dilemma whether one should keep development support away for this group. Also, politically, the government may not be in a position to keep development support for much long, and may in fact provide another similar support (say instead of milk, a calcium-fortified orange juice). So, what happens in the latter case?

Submitted by Raymond on
I believe RCT is one of several ways of measuring impact of an intervention. Often, we try to answer the question; what has out intervention changed? With RCT, how can the following be addressed. 1. How do you control for other changes that occur simultaneousily with your intervention. Say, beside the milk, if the government intriduces a new sylabus. 2. What is the scope, in terms of time, in a RCT?

Submitted by Anonymous on
I hoped I would find some more interesting information in this piece of blog. I suppose that no one was infering that RCT would reduce poverty, only that they can help measure the effect on poverty of a given intervention. Yes, the intervention should be relevant and responding to a real need. Who said (and why) that RCT don't say anything about what happens if a government is implementing a program? Why can't RCT be administered on a gvt-implemented program? I know of bank-financed (and gvt-implemented) projects that use randomized design. It's kind of frustrating to be told that you cannot measure impact if you don't have control groups and then that RCT are kinda worthless anyway.

Thanks for your thoughtful comment. RCTs can evaluate government programs (Progresa in Mexico was a government program). But the bulk of the experiments discussed in the Glennerster-Kremer paper were not government programs, but rather experiments (run by NGOs) to test individual behavioral responses. Nevertheless, RCTs can be extremely valuable to governments if they were to assess different types of delivery mechanisms or different institutional arrangements. For instance, you could compare delivering education through a voucher scheme with one where the central government finances and provides education. These different delivery mechanisms often address the "government failure" associated with real-world public policies, so RCTs that shed light on overcoming such failures could make a major contribution.

Submitted by olugbenga adesanya on
Right you are. I hope governments in African nations would adopt the RCT.

Submitted by Anonymous on
Randomization ONLY identifies the mean treatment effect. That is, what happened on average. Without further assumptions, it is impossible to identify how many people benefited from treatment and which people benefited the most. If a medical intervention improved the health of people in a village on average, that's great on a surface level. However, with heterogeneity in the treatment response it could be the case that those with the worst health did WORSE from the treatment, and those with the best health did the BEST from the treatment, and their effect offset the losers in magnitude. No policy maker would advocate such an intervention.

Submitted by Chike on
Is there a blog where non economists / development specialists can understand what is going on with respect to the World Bank activities in Africa. The subject matter of this blog seems to be very specialised.

Thanks for your question, Chike. You can find out about the World Bank's activities in Africa on www.worldbank.org/africa. Incidentally, this particular post on randomized control trials is specialized, but many of the others are not, so I hope you continue to visit Africa Can. Regards, Shanta

A major point we can take from here like you maintained is 'RCTs tell us what will happen if you intervene; they don’t answer the question: “Should you intervene?” The answer lies in whether there is a market failure or need for redistribution' Until further evidence is supplied, we may still remain on same spot. Anyway it worths the time invested

Submitted by Imran on
I agree that while many economic theories are impractical to implement in reality, they still allow us to see where the bar can be set to in a stylized world. Then the question becomes why can't we get there from where we are today? RCT are no different. Yes, they may be difficult to implement, there may be obstacles and opportunity cost associated with the resourced need to carry them out. However, despite these limitations the information they provide is critical. It is important to know what magnitude of difference a "treatment" can make. Then decide why it may or may not to be possible to implement on a larger scale (i.e. from NGO to Governemnt).

RCT are a very valuable tool for policymakers (and decision makers in general). They offer valuable information. But this information needs to be used along side a other types and must be carefully considered before taking action. The hype of RCT should not lead us to forget that we have been here before. The foundation of modern think tanks in the early 1900s was fueled by a belief that scientific method could cure society of it's ills. Driven by metaphors from medicine (cure the patient), physics (make society efficient), engineering (plan and design) and even religion (build new states) generations of new think tanks and research programmer have followed each other in this quest. And although they have made great contributions they have failed to achieved their visions. Most forgot that societies are not machines nor patients who can be told what to do. Ideology matters and the manner in which decisions are made is sometimes far more important than the decisions themselves. Donors have been quick to jump on the bandwagon of this new gold standard and are now promoting far reaching reforms driven by 'evidence' but allowing little debate among those whose lives wil be affected by these changes. The rush to achieve impact leads to undermining political debate, weakens political parties who watch idly while their governments broker deals with donors for policies that can change the very nature of their society -cash transfers or user fees are not technical issues: they are Political. It would be great if these initiatives promoted a debate on the nature of decision making like the one that nations such as India and China have through the promotion of new generation of think tanks (see http://onthinktanks.org for some examples from China)

Submitted by Anonymous on
Of course RCTs can reduce poverty. Hire poor students from a local university as enumerators and there you are. I guess the point I am trying to make is that just as with Blogs, the usefulness of RCTs depends on people asking the right questions and making sure that their tools can actually be used to answer them. Yes, particularly RCTs of NGO interventions are typically used to peddle a pre-conceived point of view. A great example of what happens when people fundamentally don't understand what the shiny tool they are using is capable of is Esther Duflo’s TED talk where every one of the three examples she presents was distorted in one way or another to confirm previously held beliefs. But none of this means that RCTs can’t be really useful and value for money.

Submitted by Anonymous on
Check out the One Acre Fund in E. Africa for an example of a more ethical approach to RCTs where control groups become participants. This limits long-term longitudinal comparison but is certainly more ethical.

Add new comment