A key issue in any impact evaluation is take-up (i.e. the proportion of people offered a program who use it). This is particularly an issue in many finance and private sector (FPD) programs. In many health and education programs such as vaccination campaigns or getting children to school programs, the goal of the program is actually to have all eligible individuals participate. In contrast, universal take-up is not the goal of most FPD programs, and, even when it is a goal, it is seldom the reality. Not all households or firms will want or need a loan, wish to purchase insurance, or desire to participate in some training program. Thus some microfinance experiments have found take-up rates of only 5-10%, and experiments with rainfall insurance likewise have struggled to get high take-up.
It is well-known that low take-up reduces statistical power (i.e. our ability to reject the null hypothesis that a program has no effect when this null is false - or in layman’s terms, whether we will be able to detect whether our program has had an effect). However, I frequently get asked how to do power calculations in this case and my experience is that many people don’t have a sense of just how much damage low take-up can do to your power calculations. So I thought I’d explain it here.
Let p be the difference in the proportion of the treatment group that takes up a given intervention relative to the control group. Then looking at the formula for the minimum detectable effect in a single cross-sectional follow-up survey (see e.g page 33 of the randomization toolkit), one can quickly see that the needed sample size is inversely proportional to the square of p.
What does this mean in practice?
· If p = 0.75 (e.g. 0% of the control group take the intervention and 75% of the treatment group do), the sample size needed is 1/(0.752) = 1.77 times as large as it would be with 100% compliance.
· If p = 0.50, the sample size needed is 4 times as large as with 100% compliance
· If p=0.25, the sample size needed is 16 times as large as with 100% compliance
· If p = 0.10, the sample size needed is 100 times as large as with 100% compliance!
· If p = 0.05 (as in some microinsurance/microfinance), you need 400 times the sample as with 100% compliance!
You see very quickly that low take-up massively increases the sample you need to detect a desired effect.
To make this crystal clear, consider an example of a business training intervention, where the goal is to increase business profits from a baseline value of $1000, assuming the standard deviation of profits is also $1000.
To detect a 10% increase in profits with p=1.0 (i.e. 100% of those in the treatment group get training, and 0% in the control group do), in STATA use the command:
Sampsi 1000 1100, sd1(1000) power(0.9)
And you will get output telling you that you need a treatment group size of 2102
Now suppose instead that you expect only half of those you invite to training to show up; and that the training then still leads to a 10% increase for those you actually train. Then the mean profits for the treated group will be 1000 + 0.5*100 = 1050, and so:
Sampsi 1000 1050, sd1(1000) power(0.9)
gives a needed treatment group size of 8406, 4 times the size you would need with 100% compliance.
So what can you do when you face this problem? I am sure this will be a recurring theme on this blog, with one of my earlier posts already touching on this. We will discuss some of these in more detail in future posts, but basically your options are to do one or more of the following:
- Increase your baseline sample (and thus expense) to deal with this, and proceed as normal.
- Collect a lot more time periods of data on each observation (see e.g. This paper)
- Try and understand why take-up is low, and undertake actions (e.g. advertising, encouragement) to try and increase take-up rates
- Restrict the study to a group of units for whom take-up would be much higher – e.g. a business training program could be advertised to all eligible firms, and then the number of slots available in the program could be randomly allocated amongst only those firms who apply.
Join the Conversation