This is a relatively small point, but one that has come up several times in conversations in the last few months, so I thought it worth noting here.
Context: you are randomly selecting people for some program such as a training program, transfer program, etc. in which you expect less than 100% take-up of the treatment from those assigned to treatment. You are relying on an oversubscription design, in which more people apply for the course/program than you have slots.
My recommendation: instead of randomly allocating people to treatment and control, randomly allocate them to three groups: Treatment, Control, and Waitlist. Depending on your expectations of the likely drop-out from the treatment group, and how many spare observations you have, you can make the waitlist group larger or smaller.
Then when the first day of training arrives and not all of those in the treatment group show up, the people running the program can turn to the waitlist to fill in the empty spots in their class. They are free to do this in any order – i.e. they can call the people who live closest to the course, or just rely on whoever answers their phone first, etc. This waitlist group then is no longer part of your experiment, but helps the people running the program meet their target/quota for the number of people trained or served.
Why do this? I’ve found through painful experience that if you don’t do this, there can be a huge temptation for the people delivering the program to dip into the control group. This can either because they have been contracted to train or deliver the program to a certain number of people, so that every empty seat directly costs them, or just because they see they have space and genuinely want to help as many people as possible. The waitlist allows them to do this without cutting into your control group, thereby maximizing the difference in program take-up rates between treatment and control.
Examples:
(Note: I am abstracting here from the possibility of randomly reassigning some of the control group to the treatment group. This is even better to do than having a waitlist if there is time to do so, but my experience has been that a lot of these issues arise when there is very little time to organize replacements, so the people offering the program want to have a group they can offer the program to on a convenience basis to meet their targets).
Context: you are randomly selecting people for some program such as a training program, transfer program, etc. in which you expect less than 100% take-up of the treatment from those assigned to treatment. You are relying on an oversubscription design, in which more people apply for the course/program than you have slots.
My recommendation: instead of randomly allocating people to treatment and control, randomly allocate them to three groups: Treatment, Control, and Waitlist. Depending on your expectations of the likely drop-out from the treatment group, and how many spare observations you have, you can make the waitlist group larger or smaller.
Then when the first day of training arrives and not all of those in the treatment group show up, the people running the program can turn to the waitlist to fill in the empty spots in their class. They are free to do this in any order – i.e. they can call the people who live closest to the course, or just rely on whoever answers their phone first, etc. This waitlist group then is no longer part of your experiment, but helps the people running the program meet their target/quota for the number of people trained or served.
Why do this? I’ve found through painful experience that if you don’t do this, there can be a huge temptation for the people delivering the program to dip into the control group. This can either because they have been contracted to train or deliver the program to a certain number of people, so that every empty seat directly costs them, or just because they see they have space and genuinely want to help as many people as possible. The waitlist allows them to do this without cutting into your control group, thereby maximizing the difference in program take-up rates between treatment and control.
Examples:
- We did this for a large-scale vocational training program in Turkey, reported here: Training providers were then asked to select a list of potential trainees that was at least 2.2 times capacity…Thus if a course had capacity for 50 trainees, and 120 were deemed eligible, 50 would be randomly assigned to treatment, 50 to control, and 20 to a waitlist.
- This was not done in Card et al’s vocational training program in the Dominican Republic, and at least one-third of the control group ended up being offered treatment.
- This was not done in a planned experiment in Uganda I had to abandon: only half of those offered the trained attended, and other firms from the experimental control group ended up being invited to make up the target number to be trained.
(Note: I am abstracting here from the possibility of randomly reassigning some of the control group to the treatment group. This is even better to do than having a waitlist if there is time to do so, but my experience has been that a lot of these issues arise when there is very little time to organize replacements, so the people offering the program want to have a group they can offer the program to on a convenience basis to meet their targets).
Join the Conversation