Should you oversample compliers if budget is limited and you are concerned take-up is low?

|

This page in:

My colleague Bilal Zia recently released a working paper (joint with Emmanuel Hakizimfura and Douglas Randall) that reports on an experiment conducted with 200 Savings and Credit Cooperative Associations (SACCOs) in Rwanda. The experiment aimed to test two different approaches to decentralizing financial education delivery, and finds improvements are greater when Saccos get to choose which staff should be trained rather than when they are told to send the manager, a loan officer, and a board member.

One point of the paper that I thought might be of broader interest to our readers concerns the issue of what to do when you only have enough budget to survey a sample of a program’s beneficiaries, and you are concerned about getting enough compliers.

The problem
The experiment randomized 65 Saccos to get training where they could select who they send (autonomous selection), 65 Saccos to get training of specific people in the organization (fixed selection), and then 70 Saccos to be the control. Treated Saccos would then try to implement this training by training members.

Each of the Saccos has an average of 5,500 members. So the 130 treated Saccos have around 715,000 members, and the 70 control Saccos 385,000. The authors are interested in measuring outcomes at the member level, and have budget to survey 4,000 out of the 1,100,000 members, or 20 members per Sacco.

The authors were concerned that the take-up rate might be low for this financial education training offered by the Sacco, so were worried that if they just randomly sampled 20 out of the 5,500 members from a Sacco, by chance they could end up with very few people who had actually completed training. They did not have a baseline survey, and did not know the take-up rate until they went to the field for their first follow-up survey.

What did they do?
For both treatment and control Saccos, the authors got a roster of all community members. For the control Saccos they simply randomly chose 20 respondents to interview. For the treated Saccos, they matched these rosters to the attendance sheets for the financial education training sessions, and then, for each Sacco, randomly selected 10 trained members and 10 untrained members per Sacco. The actual attendance rate was 21-23%, so this meant they over-sampled compliers relative to non-compliers. In their analysis, they then weight the data using the probability of being selected.

Is this optimal?
The approach used by the authors will yield correct estimates, but does not maximize power. Let C(i) be the number of compliers in strata i, SC(i) the standard deviation of the outcome for compliers, N(i) be the number of non-compliers in strata i, SN(i) the standard deviation of the outcome for non-compliers, and n be the total number of units to be sampled from strata i. Then the optimal number of units to sample from the compliers is:


Now they do not know in advance the standard deviation of the outcomes, and so if we assume that the standard deviations are equal in the two groups, then this reduces to proportional allocation – i.e. if 23% of members in a Sacco took up financial education training, then 23% of the sample of 20, or 5 people, should come from the compliers, and the remaining 15 should be non-compliers. Proportional sampling within strata then has the advantage of making the overall sample self-weighting.

Note that in their case there is an additional level of sampling stratification, which is at the level of the Sacco. This will mean that the optimal proportion of the sample of 20 that should be compliers will then vary across Saccos – Saccos with higher take-up rates will have more trained members interviewed, while those with lower take-up rates will have fewer members interviewed.

So why might you want to oversample compliers?
The formula above suggests that the best thing to do is stratify on compliance, but then sample in proportion to the take-up rate. However, it does give one possible reason to oversample compliers – if you think the treatment will increase the variation in outcomes among those treated, then you will want to sample relatively more compliers.

In discussing this with Bilal, he mentioned a more practical reason: because they did not know the take-up rate in a community until they arrived at a Sacco, the field team had to match the attendance list to the Sacco roster, and then draw the sample. The simple rule of randomly choose 10 trained and 10 untrained to interview then involved much less enumerator calculation and possible discretion than having different sampling fractions in every community that had to be calculated on the spot.

Two other reasons you might want to oversample compliers can come about if compliance is really low:

  • You might be interested in doing take-up regressions to help understand what characteristics are associated with take-up, and so want to make sure you have sufficient numbers of compliers to compare the non-compliers to.
  • You might be concerned that take-up rates are so low that you may need to fall back on alternative non-experimental methods that focus on the impact on compliers, as in this other financial education evaluation.
A couple of final points
  1. I’ve talked here about over-sampling compliers, but of course you may also want to over-sample non-compliers if take-up rates are very high and yet you want to say something about who doesn’t take up a program, or if non-compliance increases variance.
  2. Another example of over-sampling compliers comes up in the Miracle of Microfinance paper – they take a two-step procedure to deal with low take-up concerns. First, they conduct a census and identify characteristics associated with take-up. They then restrict their population of interest to a subset of households that have higher take-up – those who had lived in the area at least 3 years, and that had a women aged 18 to 55. They then note “Spandana borrowers identified in the census were oversampled because we believed that heterogeneity in treatment effects would introduce more variance in outcomes among Spandana borrowers than among nonborrowers, and that oversampling borrowers would therefore give higher power. The results ... weight the observation to account for this oversampling so that the results are representative of the population as a whole”.  However, they do not report how much oversampling took place, or how they decided how much more variance they thought might occur.

Authors

David McKenzie

Lead Economist, Development Research Group, World Bank

Join the Conversation

JP
September 17, 2018

Hi there, beginner with sampling methods here and not an economist...can you please explain why they didn't just use the attendance sheets as the sampling frame and randomly select from trained members? Is the answer "Because that wouldn't preserve the original random assignment?"

David McKenzie
September 18, 2018

Hi JP,
You are correct, what is randomly assigned is the Offer of training, not whether or not people decide to take it. So the authors need to compare a random sample of those offered training (the treatment group)  to a random sample of those not offered training (the control group). Comparing the means for these two groups gives what is called the intention-to-treat effect. If you divide this by the proportion of those offered training who actually received it, you can also recover what is called the treatment effect on the treated - the effect of actually receiving training for those who take it up when offered. But both require surveying not only those who attend the training, but also those who are offered training but don't attend.

Nicholas Owsley
February 28, 2019

Hi David, if your primary goal was to determine an ATT for a cluster-randomized trial, would you then use the sampling method used in the Miracles of Microfinance (quoted above), where you predict take-up and thereby sample only individuals with a high probability of take-up in each of your control and treatment villages (to reduce cost/increase power) and then use an IV approach to determine the effect within this sample, with random assignment to treatment village as the IV?

David McKenzie
February 28, 2019

This is the approach they use, and can be appropriate in some settings. There are three scenarios:
1) You have baseline data for everyone, but only budget to survey a subset at follow-up - then the above approach can be used to help choose who to survey.
2) You have baseline data for a sample, but then think the proportion who take-up will be low, so would like to have a different sample for follow-up. You can then use your baseline sample and take-up data to predict take-up, and then a screening questionnaire to select into the follow-up sample. This is effectively what they did. The big issue here is making sure you only use time-invariant characteristics to predict take-up (since you won't have baseline data for some people you want to survey); and a big concern that people might migrate out of (or into) your village, so that those you survey only at follow-up might lead to bias if the treatment affects migration.
3) You don't have baseline data, and are deciding who to survey for follow-up - but you have data from the program admin records on taker-uppers, and can use that to see that e.g. compliers are more likely to be female and aged 25-40, and so then have a screening questionniare which oversamples these individuals. Again the concerns are about migration and other ways you miss people.