Syndicate content

Ethical Validity Response #2: Is random assignment really that unacceptable or uncommon?

David McKenzie's picture
In his post this week on ethical validity in research, Martin Ravallion writes:
 “Scaled-up programs almost never use randomized assignment so the RCT has a different assignment mechanism, and this may be contested ethically even when the full program is fine.”

Lotteries aren’t so exotic
But there are plenty of examples where random assignment is used in programs, not at the bequest of researchers, but because this is seen as a fair way to allocate spaces in a program. This particularly struck me given that this year alone I’ve entered into lotteries to get into the White House Easter Egg Roll, the New York Marathon, the Marine Corps Marathon, and the Cherry Blossom 10 miler (so far I’m 2 for 3). None of these are randomized because a researcher wanted them to be (at least as far as I’m aware), but because this is seen as a fair way of allocating spaces.

Ok, you say, but rolling Easter Eggs and running through large cities aren’t real policies that people care about. Well, I’ve also been through lotteries to get my kids into their language immersion program at school, and years ago tried (unsuccessfully) the US Diversity Visa (Green Card Lottery).  So I’ve also participated in more major lotteries.

Indeed lotteries in education seem increasingly common in the U.S., particularly for admission for charter schools. Equity and transparency are key reasons given for this – indeed the DC Public Charter schools run lotteries for admission, with their website saying “To ensure that children in the District of Columbia receive fair and equitable opportunities to enroll in and attend public charter schools, the District Of Columbia Public Charter School Board (PCSB) has created enrollment and lottery guidelines.”. Indeed all schools receiving Federal Charter School Program funds must use a lottery if there are more applicants than places. Other examples of lotteries used in the U.S. are (famously) for the Vietnam military draft, for the expansion of Medicaid in Oregon, many colleges use lotteries to assign dorms or roommates, lotteries are used for some national park programs (including a road lottery and an alligator lottery!) –as well as running races, Olympic tickets, Superbowl tickets, etc. all being allocated at least in part by lottery. So the idea that people find lotteries strange and unethical doesn’t concord with their use in a whole range of different scenarios.

But can’t we do better than lotteries?
Martin argues that random assignment may not be the most ethical way to allocate scarce resources when there is excess demand, suggesting that participants instead be chosen in terms of who is likely to benefit most. This is certainly possible in some cases, but I have two concerns with it.

First, I am very skeptical that we know well who is going to benefit most from most programs. The problem is that different theories can often give opposing predictions, and it is only through prior evaluations that we may start to form some sense of who programs might work best for. Consider his example of a training program. I’ve evaluated a number of these programs, and it is never clear whether we should expect it to benefit most those with initially higher levels of skills (based on the idea that training complements existing skills) or those with initially lower levels of skills (based on the idea that it substitutes for existing skills). Similarly it is often easy to come up with theories why it may work better for men than women or vice versa, for poor than rich or for rich than poor, etc. As I’ve discussed elsewhere, identifying levels is very different from identifying who will benefit most from a treatment – communities might be quite good at identifying who is poor, but I suspect much less good at identifying who will gain most from receiving some program (it may not be the poorest).

This is where I think some people perhaps read more into this paper by Alatas et al. (2012) that Berk referred to than I do. My understanding is that they show getting communities to pick who is poor does almost as well as using a proxy means test in terms of approximating per capita expenditure, and the process seems to make people happier than using a proxy means test. But while it tells us communities may have information that helps pick who is poor, it tells us nothing about whether communities are good at picking who will benefit most from a program. Even if the program is as simple as giving people cash, it need not be the case that choosing the poorest, most marginalized individuals results in the greatest lasting impact – as seen for example in my work with Suresh de Mel and Chris Woodruff giving cash grants in Sri Lanka.

Second, I am skeptical that the alternative to random assignment in many of the places we work is technocratic assignment based on who will gain most
in many cases the alternative is selection based on who you know, or first-come first-served (which can benefit those with the lowest transaction costs or largest information networks). Martin asks us to imagine a government telling its underserved citizens that the reason they didn't get roads was because of randomization; this strikes me as better than saying the reason you didn't get them is because another district's congressman sat on the appropriations committee and made sure all the new roads being built ended up in his or her district.  

I think this is certainly an area where more research is of interest – I’d like to see more work which compares treatment effects under different assignment rules. I’m sure in some cases one can do better than (conditional) random assignment, but suspect such cases are relatively rare. Just as there have been arguments that cash transfers are a useful benchmark to compare aid programs against; (conditional) random assignment seems like a useful benchmark allocation mechanism, with the onus then on those claiming they can do better to show this is the case.
 

Comments

Submitted by Paul Harvey on

This is a striking case of 'great minds think alike / someone else always has your good ideas first'. Kevin Savage at World Vision and I were literally yesterday trying to come up with good ideas for a 3ie workshop on impact evaluation in relation to humanitarian aid. What we were talking about is that targeting is usually done so poorly that it seems ripe for some experimental research. We were trying to come up with a design that would compare lottery, choosing poor/ vulnerable people administratively with proxy indicators and asking communities to select the poorest. To get around ethical concerns in humanitarian crises I was suggesting you could run an experiment where different groups were provided with an additional cash bonus. I'll let you know if we get anywhere with turning a conversation on a ski chair lift into a proper proposal and getting it funded.

Submitted by Nate Barker on

Hi Paul,

You might find some research that Dean Karlan and Bram Thuysbaert have done to be especially relevant. Here's a working paper that explores that question:
http://www.nber.org/papers/w19646.pdf
They look at various approaches to selecting the poorest households (participatory rural appraisals, proxy means test, proxy means + verification), and compare the results against a more comprehensive survey on consumption. The main takeaway is that while all of those methods perform reasonably well at picking the poorest households and excluding the wealthiest households, beyond that, none of the options do especially well. My takeaway from this research is that a more parsimonious approach to targeting is likely better in terms of cost-effectiveness. It also presents a fairly strong case for randomizing within a fairly large pool of potentially eligible individuals.

The other key point to note that I try to make in the blog post is that it is a different thing to try and identify who are the poorest households than it is to identify who is going to respond best to some intervention - particularly when it is a new intervention that people haven't seen before.

Submitted by Nate Barker on

Right, I agree completely that the two are often very different things. I think though that this point raises another ethical question (and one to which I don't know that there's a clear answer): is the "best" program one that successfully targets the people most likely to be successful, or that targets the most needy? For example, in your evaluation of a cash/in-kind transfer to micro-enterprises in Ghana (with Chris Woodruff, Marcels Fafchamps and Simon Quinn) you find that when given grants, women with initial profits below the median are on average not profitable when given the transfer. In contrast, it is the wealthier households who tend to benefit the most. However, I'm not sure from that that the logical conclusion is that free in-kind transfers should only be given to wealthier households. I think there's perhaps a case to be made that poorer households are equally deserving of such an intervention, even if they are on average less likely to be able to use it successfully.

Given how little we often know about who will be most successful, I think the hypothetical scenario I raise only affects a limited number of interventions. However, even in an ideal world where we know who is most likely to benefit from a program, I'm not entirely certain that should be our primary method of targeting.

Add new comment