Syndicate content

Tips for Randomization in the Wild: Adding a Waitlist

David McKenzie's picture
This is a relatively small point, but one that has come up several times in conversations in the last few months, so I thought it worth noting here.
Context: you are randomly selecting people for some program such as a training program, transfer program, etc. in which you expect less than 100% take-up of the treatment from those assigned to treatment. You are relying on an oversubscription design, in which more people apply for the course/program than you have slots.

My recommendation: instead of randomly allocating people to treatment and control, randomly allocate them to three groups: Treatment, Control, and Waitlist. Depending on your expectations of the likely drop-out from the treatment group, and how many spare observations you have, you can make the waitlist group larger or smaller.

Then when the first day of training arrives and not all of those in the treatment group show up, the people running the program can turn to the waitlist to fill in the empty spots in their class. They are free to do this in any order – i.e. they can call the people who live closest to the course, or just rely on whoever answers their phone first, etc. This waitlist group then is no longer part of your experiment, but helps the people running the program meet their target/quota for the number of people trained or served.

Why do this? I’ve found through painful experience that if you don’t do this, there can be a huge temptation for the people delivering the program to dip into the control group. This can either because they have been contracted to train or deliver the program to a certain number of people, so that every empty seat directly costs them, or just because they see they have space and genuinely want to help as many people as possible. The waitlist allows them to do this without cutting into your control group, thereby maximizing the difference in program take-up rates between treatment and control.

Examples:
 
  • We did this for a large-scale vocational training program in Turkey, reported here: Training providers were then asked to select a list of potential trainees that was at least 2.2 times capacity…Thus if a course had capacity for 50 trainees, and 120 were deemed eligible, 50 would be randomly assigned to treatment, 50 to control, and 20 to a waitlist.
  • This was not done in Card et al’s vocational training program in the Dominican Republic, and at least one-third of the control group ended up being offered treatment.
  • This was not done in a planned experiment in Uganda I had to abandon: only half of those offered the trained attended, and other firms from the experimental control group ended up being invited to make up the target number to be trained.
This is particularly important when you are dealing with large government programs that occur in many locations. When you are dealing with a small NGO program that is being overseen by your field coordinator, you might be able to maintain enough oversight to stop the control group getting invited. But once invites are decentralized into lots of locations, you need a rule that can make it easy for the partners to meet their obligations for beneficiary numbers, while helping also ensure your evaluation design offers the maximum power possible.

(Note: I am abstracting here from the possibility of randomly reassigning some of the control group to the treatment group. This is even better to do than having a waitlist if there is time to do so, but my experience has been that a lot of these issues arise when there is very little time to organize replacements, so the people offering the program want to have a group they can offer the program to on a convenience basis to meet their targets).
 

Comments

Submitted by Steve Glazerman on

Randomization to waitlist should be routine.
Thanks for posting this. It reminds me that I should add our waitlist protocol to the trial registry for a field experiment in Nicaragua, where we've been doing exactly this.

It's important that implementers not manage the waitlist themselves. If they know the identity of the sample member who is next in line, it could influence behavior in ways that affect outcomes. Our preference is for a member of the research team to provide names as we are notified of each vacancy that needs to be filled.

I've done researcher-managed rolling random assignment and wrote about it here (http://www.mitpressjournals.org/doi/abs/10.1162/EDFP_a_00059#.VRlMu-HG8b4). It's a good complement to batch assignment when randomization must be completed before rosters of participants are finalized and locked in stone.

Submitted by Ambrish on

Dear David,

I am faced with the following situation:

Courses (Electrical work and air condition repairs) are to start in June 2015. Duration is three months. Eligible applicants are divided into treatment, waitlist and control groups. This is all fine. Idea is to assess the short-term and long-term impacts.

But those who are now in the control group can not be prevented from applying again 3 months down the line when new batch/ course begins. This implies that a portion of current control/ waitlist group might actually get enrolled in the courses.

How do I think of impact evaluation? Your thoughts?

Thank you so much,

There are two issues here:
1) How to ensure you don't lose your whole control group by having them win in the next drawing;
and
2) How to deal with multiple lotteries in the analysis.

For the second point I blogged about this a while back:
http://blogs.worldbank.org/impactevaluations/tools-of-the-trade-dealing-with-multiple-lotteries

On the first point, it really depends how popular your program is - if you have 1000 applicants for 200 positions, then even if your control group reapplies in the next round, most of them still won't win, and you are fine provided you deal with point 2). If on the other hand there isn't much demand for the program, and everyone in the control group wins in the second drawing, you are much more stuck. Then standard methods to try to boost application rates are your best approach.

Submitted by Ambrish on

Thanks a ton, David.

I will take a look at this.

Add new comment