The importance of study design (why did a CCT program have no effects on schooling or HIV?)

A recent paper in Lancet Global Health found that generous conditional cash transfers to female secondary school students had no effect on their school attendance, dropout rates, HIV incidence, or HSV-2 (herpes simplex virus – type 2) incidence. What happened?

A careful read of the paper (and the comment that was published alongside) points to a lot of pitfalls that could have been avoided at the outset:

• The study took place in an area in South Africa, where 80% of the households in the study sample at baseline was receiving the monthly child support grant, which was approximately $30 per month at the time and also equal to the per capita monthly household expenditure. Assuming a household size of 5, that is equivalent to about 20% of monthly household expenditures, putting the CSG amount alone at the higher end of cash transfer programs as a share of household expenditures. The intervention doubled this amount, conditional on 80% school attendance. There is likely to be a big difference in schooling responses when a household goes from receiving nothing to $30 a month vs. from $30 to $60…
• The eligibility crietria for study participation was as follows: “We included girls aged 13–20 years if they were enrolled in school grades 8–11, not married or pregnant, able to read, they and their parent or guardian both had the necessary documentation necessary to open a bank account, and were residing in the study area and intending to remain until trial completion.” That is a far cry from 13-20 year-old females and, with every added criterion, goes away from including the most vulnerable to school dropout, teen pregnancy, and STIs. Intending to remain until trial completion? Other than expense, what is wrong with tracking people who move?
• School attendance rates during the study period (three years) were extremely high at 95%. Dropout per year was about 3%. So, it was practically impossible for a CCT program to have any meaningful effect on schooling rates. Given that this mechanism (increased schooling leading to reduced HIV incidence) was the one the study was trying to test, that channel was shut from the outset. The investigators seem to have been caught off-guard by these high rates, but the paper suggests that enrollment rates were about 85% among 16-18 year-olds in this area in 2012-2013. If they were that high for the general population of 16-18 year-olds, they were bound to be higher for younger people who can read, open a bank account, and keen on staying where they are for the next three years. Furthermore, even if the enrollment rate in the control group did actually turn out to be 85%, a successful CCT program may increase that to, say, 90%. Even with a HIV incidence among drouputs that is an order of magnitude larger than found in the study (2% per person year), that would have reduced the incidence rate by 1% per person year. Again, it was unlikely that the study would have found an HIV effect through increases in schooling.
• Speaking of power, the trial was individually randomized. In a study, where the primary outcome is a rare outcome like HIV incidence, you do want all the power you can get, so individual randomization is ideal from that standpoint. But, that leaves the study exposed to the violation of “no interference” requirement for causal identification of CCT effects. The paper and the comment both invoke the possibility of Hawthorne effects to explain high levels of schooling in the control group, but equally likely are the facts that the study sample is a select group of secondary school students and that the control group may have been exposed to treatment via peer effects. Regular attendance checking and voluntary counseling and testing for HIV at baseline and every year amount to other interventions (along with the $30/month CSG) that can further dull the effects of the CCT program
• The study aimed to identify the effect of schooling on HIV, but the design of CCT vs. control would not have been able to isolate the effects of schooling, separate from that of the additional income. It would still have been OK, however, to show that CCTs can reduce HIV incidence, whatever the mechanism. But, with the very high enrollment rates found ex-post, the CCT program actually amounted to giving money to households for something they would have done on their own. So, it could be seen as a windfall for a few years, almost like a UCT. Why did that not have an effect? Likely because, the study population is not the most vulnerable to teen pregnancies, early marriage, transactional sex, and HIV infection. That would be those who drop out of school, for whom additional income (with no strings attached) could have reduced these risks. But, neither the study population contained a significant mass of this vulnerable population, nor did it offer UCTs…
• Finally, the study has large and significant differential attrition. The final analysis sample contains 1114 (of 1223) individuals in the control group and 1214 (of 1225) in the CCT group, which ends up being 91 vs. 99% retention rates. The trial profile suggests that the attrition happened very fast, at the first follow-up.

So, what could the study have done differently? They could have first conducted the study somewhere else, where the “first stage” (of keeping young females in school) had a chance of succeeding. Second, they could have then chosen a group of older females with minimal other restrictions: certainly no restrictions about literacy (or being in school: you could offer money to return to school), ability to open bank accounts, or not intending to move away. Third, they could have tried to minimize study aspects that amount to small interventions on their own, such as annual HIV counseling and testing, including at baseline, which the comment states is “…a standard of care that is unusual in the region.” Fourth, they could have minimized the rubric of education and HIV focus that surrounded all study schools, by having a trial that was cluster-randomized. Fifth, they could have given unconditional cash transfers as well. Sixth, and finally, they could have had extensive tracking protocols that would have minimized differential attrition.

Finally, what can this study tell us about the relationship between schooling and HIV risk? Not much. But, this did not stop the authors from concluding the following: “Cash transfers conditional on school attendance did not reduce HIV incidence in young women. School attendance significantly reduced risk of HIV acquisition, irrespective of study group. Keeping girls in school is important to reduce their HIV-infection risk.” The middle sentence is a simple correlation between school enrollment and HIV incidence. This is neither causal (unlike the first sentence) nor a new finding. Any data set that includes in-school and out-of-school females of the same cohort and has HIV biomarkers will show large differences between the two groups in terms of HIV incidence or prevalence. But, that does not mean that if you could somehow prevent some of those dropouts, this would lead to reductions in HIV incidence. In this paper in Lancet from 2012, Baird et al. show that massive gains in school enrollment (odds ratios close to 10) did not lead to any gains in HIV infection among females who were out of school at baseline (Table 3, Panel: Baseline dropouts), despite high risk of HIV infection among this group. In Kenya, this paper by Duflo, Dupas, and Kremer show that education subsidies led to increases in educational attainment and lower dropout rates, but did not affect HIV or HSV-2 rates. An otherwise carefully conducted RCT should not be relying on observational, secondary analysis to make statements about the causal relationship between schooling and HIV risk that may or may not be true. Income effects in conditional cash transfers are more likely to be the channel in risk reduction for a whole host of hazards facing young women rather than the conditions, which clash with the safety net aspects of these programs.

Even the comment that accompanies this paper is not immune from making statements that are speculative: “This trial had robust statistical analyses and a carefully devised hypothesis—i.e. conditioning cash transfers on education would improve school attendance and consequently reduce HIV risk. But, as the authors of this study found, conditionality might make cash transfers both less feasible and less effective.” The second part of that statement may or may not be true, but there is no evidence in this study of any such thing. The commenters argue that the study had to leave out, because of the schooling conditions, those who were out of school or who were pregnant. Except that it did not have to do that – it was a choice, likely because the study wanted the sample to be school-based and recruiting females out of school would have required listing them in the surrounding communities. But, this did not have anything with the conditionality: even if you offered UCTs to out-of-school youth, you’d still have to recruit them. Once you did, asking them to attend school was also possible.

When study findings don’t fit our priors, we need to be able to change our minds even when it is our own study…