Syndicate content

The importance of study design (why did a CCT program have no effects on schooling or HIV?)

Berk Ozler's picture

A recent paper in Lancet Global Health found that generous conditional cash transfers to female secondary school students had no effect on their school attendance, dropout rates, HIV incidence, or HSV-2 (herpes simplex virus – type 2) incidence. What happened?

A careful read of the paper (and the comment that was published alongside) points to a lot of pitfalls that could have been avoided at the outset:

  • The study took place in an area in South Africa, where 80% of the households in the study sample at baseline was receiving the monthly child support grant, which was approximately $30 per month at the time and also equal to the per capita monthly household expenditure. Assuming a household size of 5, that is equivalent to about 20% of monthly household expenditures, putting the CSG amount alone at the higher end of cash transfer programs as a share of household expenditures. The intervention doubled this amount, conditional on 80% school attendance. There is likely to be a big difference in schooling responses when a household goes from receiving nothing to $30 a month vs. from $30 to $60…
  • The eligibility crietria for study participation was as follows: “We included girls aged 13–20 years if they were enrolled in school grades 8–11, not married or pregnant, able to read, they and their parent or guardian both had the necessary documentation necessary to open a bank account, and were residing in the study area and intending to remain until trial completion.” That is a far cry from 13-20 year-old females and, with every added criterion, goes away from including the most vulnerable to school dropout, teen pregnancy, and STIs. Intending to remain until trial completion? Other than expense, what is wrong with tracking people who move?
  • School attendance rates during the study period (three years) were extremely high at 95%. Dropout per year was about 3%. So, it was practically impossible for a CCT program to have any meaningful effect on schooling rates. Given that this mechanism (increased schooling leading to reduced HIV incidence) was the one the study was trying to test, that channel was shut from the outset. The investigators seem to have been caught off-guard by these high rates, but the paper suggests that enrollment rates were about 85% among 16-18 year-olds in this area in 2012-2013. If they were that high for the general population of 16-18 year-olds, they were bound to be higher for younger people who can read, open a bank account, and keen on staying where they are for the next three years. Furthermore, even if the enrollment rate in the control group did actually turn out to be 85%, a successful CCT program may increase that to, say, 90%. Even with a HIV incidence among drouputs that is an order of magnitude larger than found in the study (2% per person year), that would have reduced the incidence rate by 1% per person year. Again, it was unlikely that the study would have found an HIV effect through increases in schooling.
  • Speaking of power, the trial was individually randomized. In a study, where the primary outcome is a rare outcome like HIV incidence, you do want all the power you can get, so individual randomization is ideal from that standpoint. But, that leaves the study exposed to the violation of “no interference” requirement for causal identification of CCT effects. The paper and the comment both invoke the possibility of Hawthorne effects to explain high levels of schooling in the control group, but equally likely are the facts that the study sample is a select group of secondary school students and that the control group may have been exposed to treatment via peer effects. Regular attendance checking and voluntary counseling and testing for HIV at baseline and every year amount to other interventions (along with the $30/month CSG) that can further dull the effects of the CCT program
  • The study aimed to identify the effect of schooling on HIV, but the design of CCT vs. control would not have been able to isolate the effects of schooling, separate from that of the additional income. It would still have been OK, however, to show that CCTs can reduce HIV incidence, whatever the mechanism. But, with the very high enrollment rates found ex-post, the CCT program actually amounted to giving money to households for something they would have done on their own. So, it could be seen as a windfall for a few years, almost like a UCT. Why did that not have an effect? Likely because, the study population is not the most vulnerable to teen pregnancies, early marriage, transactional sex, and HIV infection. That would be those who drop out of school, for whom additional income (with no strings attached) could have reduced these risks. But, neither the study population contained a significant mass of this vulnerable population, nor did it offer UCTs…
  • Finally, the study has large and significant differential attrition. The final analysis sample contains 1114 (of 1223) individuals in the control group and 1214 (of 1225) in the CCT group, which ends up being 91 vs. 99% retention rates. The trial profile suggests that the attrition happened very fast, at the first follow-up.
So, what could the study have done differently? They could have first conducted the study somewhere else, where the “first stage” (of keeping young females in school) had a chance of succeeding. Second, they could have then chosen a group of older females with minimal other restrictions: certainly no restrictions about literacy (or being in school: you could offer money to return to school), ability to open bank accounts, or not intending to move away. Third, they could have tried to minimize study aspects that amount to small interventions on their own, such as annual HIV counseling and testing, including at baseline, which the comment states is “…a standard of care that is unusual in the region.” Fourth, they could have minimized the rubric of education and HIV focus that surrounded all study schools, by having a trial that was cluster-randomized. Fifth, they could have given unconditional cash transfers as well. Sixth, and finally, they could have had extensive tracking protocols that would have minimized differential attrition.

Finally, what can this study tell us about the relationship between schooling and HIV risk? Not much. But, this did not stop the authors from concluding the following: “Cash transfers conditional on school attendance did not reduce HIV incidence in young women. School attendance significantly reduced risk of HIV acquisition, irrespective of study group. Keeping girls in school is important to reduce their HIV-infection risk.” The middle sentence is a simple correlation between school enrollment and HIV incidence. This is neither causal (unlike the first sentence) nor a new finding. Any data set that includes in-school and out-of-school females of the same cohort and has HIV biomarkers will show large differences between the two groups in terms of HIV incidence or prevalence. But, that does not mean that if you could somehow prevent some of those dropouts, this would lead to reductions in HIV incidence. In this paper in Lancet from 2012, Baird et al. show that massive gains in school enrollment (odds ratios close to 10) did not lead to any gains in HIV infection among females who were out of school at baseline (Table 3, Panel: Baseline dropouts), despite high risk of HIV infection among this group. In Kenya, this paper by Duflo, Dupas, and Kremer show that education subsidies led to increases in educational attainment and lower dropout rates, but did not affect HIV or HSV-2 rates. An otherwise carefully conducted RCT should not be relying on observational, secondary analysis to make statements about the causal relationship between schooling and HIV risk that may or may not be true. Income effects in conditional cash transfers are more likely to be the channel in risk reduction for a whole host of hazards facing young women rather than the conditions, which clash with the safety net aspects of these programs.

Even the comment that accompanies this paper is not immune from making statements that are speculative: “This trial had robust statistical analyses and a carefully devised hypothesis—i.e. conditioning cash transfers on education would improve school attendance and consequently reduce HIV risk. But, as the authors of this study found, conditionality might make cash transfers both less feasible and less effective.” The second part of that statement may or may not be true, but there is no evidence in this study of any such thing. The commenters argue that the study had to leave out, because of the schooling conditions, those who were out of school or who were pregnant. Except that it did not have to do that – it was a choice, likely because the study wanted the sample to be school-based and recruiting females out of school would have required listing them in the surrounding communities. But, this did not have anything with the conditionality: even if you offered UCTs to out-of-school youth, you’d still have to recruit them. Once you did, asking them to attend school was also possible.

When study findings don’t fit our priors, we need to be able to change our minds even when it is our own study…
 

Comments

Submitted by Audrey Pettifor on

Hi Berk
Thank you for your commentary. I certainly will say in this case hindsight is 20/20. When we first started designing this trial in 2008/2009 we knew a few things: 1) there was a pretty strong evidence base that CCTs increased school attendance, esp for girls; 2) our formative research in South Africa showed that completing high school was associated with lower HIV prevalence in young South African women (we can debate how that is not the same at attendance or incidence); 3) we were working in a very poor area of South Africa; 4) that formative qualitative work suggested that financial barriers were significant contributors to girls dropping out of school and 5) school attendance among 16 and older girls was around 80%.

So things we DID debate beforehand:
1) Is South Africa the right place to do a CCT trial based on school attendance? Yes attendance and enrollment are very high in South Africa- much higher than in other parts of the region. That said there was still pretty significant drop out in the later years after mandatory schooling and schooling appeared to be protective against HIV. Given this we felt the question was still valid. Perhaps we need to consider if CCTs for schooling are only really useful in low attendance areas (what is that cutoff?)

2) Yes we knew that the area was poor but didn't actually know how many homes were getting the Child Support Grant. That said, we did think that providing cash to the adolescent girls themselves (something the CSG does not do) would make a difference. And given that the amount we gave to the girls and household member is equivalent to expenditure per capita (and the CSG) it seemed it would make a difference. Maybe it was not enough- we can discuss actually if any amount of cash is ever enough in some settings where self-esteem and fitting in are being derived from material/aspirational goods and survival sex is less prevalent.

3) Cluster randomized vs individual- yes, we did feel the individual trial offered greater power for an HIV incidence outcome BUT we also felt that the main mechanism through which the trial would impact HIV was through schooling and that the reason girls were not attending school was financial. If the reason they were not attending was financial then in theory peer effects should not have impacted attendance. We have a forthcoming paper by Rosenberg et al. showing that there were Hawthorne and selection effects in the cohort over time. We did not anticipate this. Our study did have pretty minimal contact with participants and little educational components compared to many HIV prevention trials.

4) Regarding differential retention- yes, we should have better anticipated that the control group in an individual randomized trial would not be happy about getting the control arm (that said in our pilot study people said they were ok with being randomized to the control arm). Either way, I do think some girls who we lost basically decided to drop out of the study as soon as they opened the envelope and saw that they were in the control arm. We had very intensive retention procedures (can talk about all the various things we did to make attending assessment visit attractive), followed people when they moved, dropped out, no matter the outcome and have outcomes for almost everyone originally enrolled- it was that they refused to come back. Cluster randomization might have helped reduce the differential. We did conduct inverse probability weights to account for this and the result did not change.

5) We have 3 papers in submission right now examining school attendance and HIV incidence with robust longitudinal methods. There seems to be very strong evidence that school attendance is protective against HIV and we have explored the mechanisms through which schooling is protective and the results are not surprising but I think are robust and add to the evidence base that schooling is protective.

6) the CCT did have important impacts on gender based violence and we also have a paper in submission exploring pathways through which the intervention reduced physical violence from male partners among girls getting the cash. The results are also important.

So...not sure where I need to change my mind. I still do think staying in school reduces HIV risk. That said, are CCTs the most effective intervention to reduce HIV in young girls in this area in South Africa?- probably not. And I agree that UCTs for older girls may be more effective for this purpose. Of course this is almost 10 years after we conceived of the study and have learned a lot about cash transfers and HIV risk in that time. If only we had all this knowledge when we embarked on the trial 10 years ago :)
Best
Audrey

Submitted by Berk Ozler on
Thanks for the comment.

On the issue of you changing your mind, obviously that is not necessary: given theory and other extant evidence, the findings from one extra study may do very little to change our priors about a causal relationship - that's OK. What is not OK, however, is to have a sentence on the first page of the paper (under the Interpretation sub-section of the Summary) that goes: "Keeping girls in school is important to reduce their HIV-infection risk." This does not follow from your RCT. Put in other words, if this paper can make that statement in its abstract, so can any other paper that contains data on HIV and school enrollment status. Even if the authors put that sentence in the abstract, the editor should not have allowed it. I thought that biomedical journals were much better at sticking to a reporting template and minimizing speculation.

On hindsight being 20/20, sure, that is true. However, it's hard to chalk up the large number of things that accumulated to produce the null results as coincidental and all due to back luck - some of it, such as study setting and eligibility for the target population could have certainly been adapted between conception in 2008-09 and the actual start of recruitment in 2011 (about five years prior to eventual publication). Similarly, UCTs could be part of the trial...

Finally, yes, potential impacts on gender-based violence are important. But, the paper disinuguishes itself from the rest of the literature by its focus on HIV incidence, rather than self-reported sexual behavior or HIV prevalence. Then, the findings on the secondary outcomes must be subject to the same scrutiny on self-reported outcomes elsewhere in the literature.

Sincerely,

Berk.
 

Add new comment