This came up in the context of work with Ganesh Seshan designing an evaluation for a computer training program for migrants. The training program was to be taught in one 3 hour class per week for several months. Classes were taught Sunday, Tuesday and Thursday evenings from 5-8 pm, and then there were four separate slots on Friday, the first day of the weekend. So in total there were 7 possible sessions people could potentially attend. However, most migrants would prefer to go on the weekend, and many would not be able to attend on particular days of the week. The question was then how to allocate migrants applying to the program to treatment and control in a way that allowed for their preferences, while still enabling comparison of randomized treatment and control groups.
This has some similarities to the matching work that Al Roth received his Nobel prize for, and which is used in several school lotteries in the United States. The approach often used there is known as the deferred acceptance algorithm. Abdulkadiroglu et al summarize the process as follows: Students would submit their ordinal rankings of schools, and schools have their priorities for determining students to admit. Then it is an iterative process whereby in each round, students apply to their most preferred school that has not yet rejected them, each school tentatively admits up to its capacity according to its priority order from new applicants and those it has tentatively admitted from the previous round, rejecting those not admitted. The tentative assignment becomes final when no student is rejected. This mechanism has several desirable properties, including that it is strategy-proof, meaning that students have a dominant strategy of revealing their rankings truthfully.
However, this mechanism seemed to have several drawbacks from an operational standpoint for us. In particular, our concern was in part not having enough of a control group for some classes; that asking migrants to rank order all the class slots was asking a lot and may raise expectations too much when we expected that the totality of classes would be several times oversubscribed; and that implementing this mechanism in a transparent way with the partner might not work.
Our approach
So here is what we came up with instead. We are not claiming any optimality properties, but thought it might be useful to share, and see if others have got experiences to share on how they have done this.
The application form allowed migrants to choose two slots to apply to, and to rank order these. We debated whether to allow them to choose one weekend and one weekday slot, but in the end didn’t put limits on which two slots they could choose. Then we proceeded to choose treatment and control as follows:
This has some similarities to the matching work that Al Roth received his Nobel prize for, and which is used in several school lotteries in the United States. The approach often used there is known as the deferred acceptance algorithm. Abdulkadiroglu et al summarize the process as follows: Students would submit their ordinal rankings of schools, and schools have their priorities for determining students to admit. Then it is an iterative process whereby in each round, students apply to their most preferred school that has not yet rejected them, each school tentatively admits up to its capacity according to its priority order from new applicants and those it has tentatively admitted from the previous round, rejecting those not admitted. The tentative assignment becomes final when no student is rejected. This mechanism has several desirable properties, including that it is strategy-proof, meaning that students have a dominant strategy of revealing their rankings truthfully.
However, this mechanism seemed to have several drawbacks from an operational standpoint for us. In particular, our concern was in part not having enough of a control group for some classes; that asking migrants to rank order all the class slots was asking a lot and may raise expectations too much when we expected that the totality of classes would be several times oversubscribed; and that implementing this mechanism in a transparent way with the partner might not work.
Our approach
So here is what we came up with instead. We are not claiming any optimality properties, but thought it might be useful to share, and see if others have got experiences to share on how they have done this.
The application form allowed migrants to choose two slots to apply to, and to rank order these. We debated whether to allow them to choose one weekend and one weekday slot, but in the end didn’t put limits on which two slots they could choose. Then we proceeded to choose treatment and control as follows:
- Rank the courses in order of popularity according to number 1 choices. When courses were tied, we then used the second-choices to split the tie, and then if they were still tied, then just chose the course based on numerical order.
- Then starting with the course that is most popular, we would enter everyone who has it as a number 1 or number 2 choice into its lottery and draw treatment and control.
- We then proceed to do the same for each other course. This way each applicant is in two lotteries provided they applied for two slots, and that each slot is oversubscribed.
- Then if anyone is drawn for two slots, we allocate them to the slot which is their number 1 pick.
- Then this opens up some gaps in courses where people who had listed it as a number 2 course were chosen in both lotteries. We then choose the next person in random number rank from that course to fill the gap (changing them from control to treatment). To prevent a lot of iterative looping, we do this only among those who haven’t currently been chosen for any course.
-
Then for analysis we just need to condition on which courses they applied for, but not on the rank they gave to these courses.
If a course is undersubscribed, then everyone who lists this course as their first or second choice is guaranteed to get trained. So these individuals get assigned to treatment with probability one, and are not longer part of the experiment (but still receive the training).
As an additional modification, we would add a small waitlist to each course, which as I have previously blogged about, means that if someone doesn’t show up for training, the provider can call the waitlist and fill the slot without having to dip into the control group.
Discussion
Some may quibble that people will try to strategize as to which two slots to apply to, and that Pareto improvements may be possible ex post. However, Carroll has shown that when agents have to rank k
In practice what happened was that despite claims from the partner that these classes would be heavily oversubscribed, a combination of factors resulted in much lower than anticipated application rates, and so by time we implemented this process most applicants were assigned to treatment with probability one. So we are in a re-design phase, but I thought it would be useful to share the process above and see if anyone else has dealt with this issue and share what they did?
Join the Conversation