Syndicate content

Tools of the Trade: Dealing with Multiple Lotteries

David McKenzie's picture

Random lotteries to allocate scarce slots for an oversubscribed program provide a useful tool for estimating impacts of such a program. However, an issue which can arise in practice is that there may be multiple lotteries that an individual can apply for. For example,

·         Abdulkadiroglu and co-authors (QJE paper, ungated version) study the impact of charter school attendance on student attainment in Boston. Students decide which schools to apply for, with independent lotteries – so a student that applies to multiple schools has more chance of being selected for a charter school than a student who applies only to one school.

·         In forthcoming work with John Gibson and Steve Stillman, I look at the impact of migrating from Samoa to New Zealand on household members left behind. There is one lottery each year, and so once we consider outcomes several years later, households could have potentially applied to the lottery more than one time.

·         This issue is coming up in a number of Government projects that I have recently been discussing evaluations for. For example, in a matching grant program, there is a call for proposals each six months, with lotteries planned to allocate scarce funds among eligible applicants each round. However, firms which are not selected in one round are free to choose whether to try again in future rounds.

If individuals or firms self-select into how many lotteries to apply for, then the characteristics of lottery winners may differ from those of lottery losers, since those who apply more times are more likely to win. Luckily, Abdulkadiroglu note there is an easy fix for this concern when it comes to estimation. They define an applicant’s risk set as the list of all lotteries to which the individual applied. One then adds dummy variables for each lottery entered as controls in the estimation.

To make this concrete, suppose there are 3 oversubscribed schools a student can apply to, and let dummy variables d1, d2, and d3 indicate whether the student applied to school 1, 2 and 3 respectively. Then to look at the impact of being in a charter school on a learning outcome, one estimates:

Learning outcome = a + b*Charter School + c1*d1+c2*d2+c3*d3 + e

By 2SLS, instrumenting for Charter School with variable Z, indicating whether an individual won a lottery admitting them to a charter school, using a first stage like:

Charter School = f + g*Z + h1*d1 + h2*d2 + h3*d3 + v

The odds of winning, and hence value of Z, will depend on which lotteries an individual entered, but conditional on which lotteries are entered, it is random whether or not they win. The same basic approach can be used in a panel setting.

The key to implementing this fix is to know exactly which lotteries an individual entered. This wasn’t the case in our Samoa work, where we did not have much confidence in the ability of remaining family members to know exactly in which years the household’s migrant member had entered his or her name unsuccessfully before their name was eventually drawn. We therefore have to use a combination of administrative data and data from the lottery loser sample to show robustness to this issue. Luckily this worked out okay in our case, but this is something that people should pay attention to in designing their data recording systems going forward – without thinking clearly about this ex ante, I can easily imagine surveys or administrative data merely recording whether or not applicants won a lottery or not, and perhaps the number of lotteries they entered, rather than yes/no records of entry into each lottery on offer.

Comments

In thinking about the comment that the "characteristics of lottery winners may differ from those of lottery losers" because they are frequent participants, my first thought was "whether I flip a coin one times or ten, it is still 50%". However, the outcome profile is not symmetric, big payoff vs. little -to-no cost. But this got me to wonder, why are there entrants who do not enter all lotteries? Based on what was discussed above, I would apply for all the charter school chances...why doesn't everyone?

whether I flip a coin one times or ten, it is still 50%...this is the case on any single throw, but what determines entry into a program here is whether you get at least one head - i.e. whether your name comes up in at least one lottery - so the more entries you do, the greater your chance of winning. But of course if the charter schools are miles from your home, you may not want to apply for them, so students might choose only to apply to a subset of the available lottery set.

Submitted by Anonymous on
David, I thought that the charter risk sets were the "year of application and the set of schools applied to for charters." Or, in your example: c1*d1+c2*d2+c3*d3+c4*d1*d2+c5*d1*d3+c6*d2*d3+c7*d1*d2*d3 What am I missing?

My example is a simplified version of the application used in the charter school study. The equation in my example is for the case where the risk set is only three lotteries. These could be 3 lotteries all in the same year, or the same lottery in different years, or some combination.