This is a relatively small point, but one that has come up several times in conversations in the last few months, so I thought it worth noting here.
Context: you are randomly selecting people for some program such as a training program, transfer program, etc. in which you expect less than 100% take-up of the treatment from those assigned to treatment. You are relying on an oversubscription design, in which more people apply for the course/program than you have slots.
My recommendation: instead of randomly allocating people to treatment and control, randomly allocate them to three groups: Treatment, Control, and Waitlist. Depending on your expectations of the likely drop-out from the treatment group, and how many spare observations you have, you can make the waitlist group larger or smaller.
Then when the first day of training arrives and not all of those in the treatment group show up, the people running the program can turn to the waitlist to fill in the empty spots in their class. They are free to do this in any order – i.e. they can call the people who live closest to the course, or just rely on whoever answers their phone first, etc. This waitlist group then is no longer part of your experiment, but helps the people running the program meet their target/quota for the number of people trained or served.
Why do this? I’ve found through painful experience that if you don’t do this, there can be a huge temptation for the people delivering the program to dip into the control group. This can either because they have been contracted to train or deliver the program to a certain number of people, so that every empty seat directly costs them, or just because they see they have space and genuinely want to help as many people as possible. The waitlist allows them to do this without cutting into your control group, thereby maximizing the difference in program take-up rates between treatment and control.
Examples:
(Note: I am abstracting here from the possibility of randomly reassigning some of the control group to the treatment group. This is even better to do than having a waitlist if there is time to do so, but my experience has been that a lot of these issues arise when there is very little time to organize replacements, so the people offering the program want to have a group they can offer the program to on a convenience basis to meet their targets).
Context: you are randomly selecting people for some program such as a training program, transfer program, etc. in which you expect less than 100% take-up of the treatment from those assigned to treatment. You are relying on an oversubscription design, in which more people apply for the course/program than you have slots.
My recommendation: instead of randomly allocating people to treatment and control, randomly allocate them to three groups: Treatment, Control, and Waitlist. Depending on your expectations of the likely drop-out from the treatment group, and how many spare observations you have, you can make the waitlist group larger or smaller.
Then when the first day of training arrives and not all of those in the treatment group show up, the people running the program can turn to the waitlist to fill in the empty spots in their class. They are free to do this in any order – i.e. they can call the people who live closest to the course, or just rely on whoever answers their phone first, etc. This waitlist group then is no longer part of your experiment, but helps the people running the program meet their target/quota for the number of people trained or served.
Why do this? I’ve found through painful experience that if you don’t do this, there can be a huge temptation for the people delivering the program to dip into the control group. This can either because they have been contracted to train or deliver the program to a certain number of people, so that every empty seat directly costs them, or just because they see they have space and genuinely want to help as many people as possible. The waitlist allows them to do this without cutting into your control group, thereby maximizing the difference in program take-up rates between treatment and control.
Examples:
- We did this for a large-scale vocational training program in Turkey, reported here: Training providers were then asked to select a list of potential trainees that was at least 2.2 times capacity…Thus if a course had capacity for 50 trainees, and 120 were deemed eligible, 50 would be randomly assigned to treatment, 50 to control, and 20 to a waitlist.
- This was not done in Card et al’s vocational training program in the Dominican Republic, and at least one-third of the control group ended up being offered treatment.
- This was not done in a planned experiment in Uganda I had to abandon: only half of those offered the trained attended, and other firms from the experimental control group ended up being invited to make up the target number to be trained.
(Note: I am abstracting here from the possibility of randomly reassigning some of the control group to the treatment group. This is even better to do than having a waitlist if there is time to do so, but my experience has been that a lot of these issues arise when there is very little time to organize replacements, so the people offering the program want to have a group they can offer the program to on a convenience basis to meet their targets).
Join the Conversation
Randomization to waitlist should be routine. Thanks for posting this. It reminds me that I should add our waitlist protocol to the trial registry for a field experiment in Nicaragua, where we've been doing exactly this. It's important that implementers not manage the waitlist themselves. If they know the identity of the sample member who is next in line, it could influence behavior in ways that affect... outcomes. Our preference is for a member of the research team to provide names as we are notified of each vacancy that needs to be filled. I've done researcher-managed rolling random assignment and wrote about it here (http://www.mitpressjournals.org/doi/abs/10.1162/EDFP_a_00059#.VRlMu-HG8b4). It's a good complement to batch assignment when randomization must be completed before rosters of participants are finalized and locked in stone.
Read more Read lessDear David, I am faced with the following situation: Courses (Electrical work and air condition repairs) are to start in June 2015. Duration is three months. Eligible applicants are divided into treatment, waitlist and control groups. This is all fine. Idea is to assess the short-term and long-term impacts. But those who are now in the control group can not be prevented from applying again 3 months... down the line when new batch/ course begins. This implies that a portion of current control/ waitlist group might actually get enrolled in the courses. How do I think of impact evaluation? Your thoughts? Thank you so much,
Read more Read lessThanks a ton, David. I will take a look at this.
Thanks for these thoughts Dave. It's an old post but it's come in handy for a project of mine underway.
Thank you very much for this very helpful post. Could you please provide a reference for the note in the bottom? "Note: I am abstracting here from the possibility of randomly reassigning some of the control group to the treatment group. This is even better to do than having a waitlist if there is time to do so [...]" Thank you very much!