Having stumbled upon this development economics symposium , which took place in DC on Saturday, this paper  (recently updated) by Galiani and McEwan attracted my attention. I have noticed quite a bit of people asking whether the effects of cash transfers vary by baseline income, and this paper is the latest to take a stab at answering exactly that question.
Here is a quick summary of the paper, which examines PRAF-II in Honduras. Cash transfers were randomly assigned within 5 strata (the authors call them ‘blocks’, as in ‘block stratification’), within which conditional cash transfers were randomized across municipalities. There are multiple treatment arms, but they’re not pertinent for thispost other than the arm that provided cash transfers conditional on school enrollment and attendance as well as health clinic visits for women (pregnant or with children under 3). However, the authors report (also citing previous papers) that only the schooling condition on enrollment (and not attendance) was enforced. This is despite the fact that one of the explicit goals of PRAF-II was to improve the enforcement of conditionalities over PRAF-I. Even that may not mean much if you have to enroll in school just once a year (as opposed to every term or semester). So, it is a conditional cash transfer (CCT) with at best loosely enforced conditions. The transfers, at less than 5% of household income, are small – smaller than most others in the region (with Mexico and Nicaragua both providing more than 20%). The transfers were made three times a year and the evaluation takes place soon after the second payments were made in treatment municipalities.
The authors use data from the census on two outcomes: enrollment and work (outside or at home) for children 6-12 who have not completed 4th grade. There are some strengths to the paper that relate to the data and the study design, which are worth mentioning at this point. First, the block randomization is done on mean height for age z-scores (HAZ) of first-graders at the municipal level. So, while this is not poverty, it is likely to be highly correlated with poverty at the municipality level (although the literature on the relationship between HH income and nutritional outcomes is far from straightforward) and provides a perfect setting to examine heterogeneous impacts of cash transfer programs by baseline average nutritional status at a very granular level (the eligibility cutoff is -2.3 HAZ, indicating that the average first-grader is stunted in study municipalities). This design also avoids the criticism of ad-hoc sub-group analysis – an increasing number of studies in economics forego this type of analysis nowadays or relegate them to appendices/footnotes because it was not pre-specified.
The data, which come from the census, could also be thought of as a plus for the study – because it is coming from an independent data source not related to the intervention/evaluation. The worry with the outcome variables is that they are self-reported and we know the issues of differential reporting  in experiments that can arise from self-reports. While not having data that is tied to the study could be seen as strength, I view it more agnostically – i.e. with a good amount of worry: this is a large government program that was redesigned specifically to emphasize the conditionalities and nothing says government like having a census worker knock on your door. If you just received your second payment a month ago and your next payment for this year (and more payments next year) is pending, you may be more likely to tell a worker of the government that your eligible child is in school and not working for pay.
With those caveats, what do the authors find? They find that the mild effects of the program on increased enrollment (8 percentage points or 12% increase over a control group mean of 65% enrollment) and decreased work come entirely from the bottom two quintiles of the study. The impacts for the remaining three quintiles are much smaller and statistically insignificant. The authors draw the conclusion that targeting such programs carefully is important for cost-effectiveness (they don’t find that PRAF-II is particularly cost-effective compared with other studies). Furthermore, they don’t find any spillover effects on ineligible children or female labor supply. The authors think this could be due to the small size of the transfers, but it is also consistent with reporting bias.
In my opinion, the authors don’t go far enough, perhaps due to limited data, in interpreting their results. First, why don’t we see any effects of the program among the top three quintiles in eligible municipalities? Figures 3 and 4 make it clear that enrollment rates among these young children, at around 65-70%, leave plenty of room for improvement. Why would additional money with conditions not improve enrollment if the outcome were not bounded from above? It’s not like these three quintiles are rich: with 70 of 298 municipalities eligible for the program (23.5%), and the impacts concentrated on the bottom two quintiles of those eligible municipalities (40%), the effects are among the bottom 9% (.235 x .4) of the municipalities with respect to mean HAZ for first-graders.
The findings are actually quite similar to another paper  by Edmonds and Schady published last year. Examining a cash transfer program in Ecuador, which did not impose explicit conditions but conducted a social marketing campaign on the importance of human capital investments for poor children, they find similar impacts on schooling and paid child employment with the latter being, wait for it, statistically significant only for the bottom 9% of the population! Have we found some magic number for targeting cash transfer programs (like the 90% debt-to-GDP ratio that has been all over the news last week)? You’ll have to keep reading…
Edmonds and Schady (2012) attribute their findings to household preferences for child welfare combined with liquidity constraints and rigidities in the paid employment sector with respect to hours. In other words, parents would rather have children not working and in school, but if they don’t have enough money and cannot borrow, then they may have to unwillingly take the kids out of school and send them to work. Because paid work usually requires a minimum hours worked per day, kids can’t combine school with paid work and end up making more money than the family actually needs to get to the threshold above which they would not send their child to work (this comes from the ‘luxury axiom’ of Basu and Van (1998) that the authors invoke). So, even though the government transfer is a small percentage of foregone earnings, it is effective in getting kids back to school if it brings households above that threshold.
This raises two questions for the Honduras study: first, is something similar going on in Honduras? I wish that the authors could do more to address this issue. Second, would the same effects be seen in an unconditional cash transfer program – like a child support grant or an unconditional cash transfer program not even aimed at children? What is the relative role of the condition here? One would think that when dealing with the poorest households, the added nudge from the substitution effect associated with the schooling condition could play a role. I am afraid that without more studies like PRAF-II that also randomize the existence and the enforcement of the condition, we are unlikely to find out. Two studies comparing CCTs and UCTs are available in Africa (Malawi and Burkina Faso), the latest of which was reported here  (and briefly discusses the previous one): they both suggest some role for the schooling conditionality.
Which brings me to whether we can take away a message from Galiani and McEwan (2013) and Edmonds and Schady (2012) that cash transfer programs should be targeted to the poorest of the poor. This would not be wise everywhere even after abstracting from the question of the main purpose of the transfer scheme. I looked at our data underlying this paper , to see if we see a similar pattern in Malawi: we don’t. Whether I examine enrollment or test scores, the impacts are pretty uniform across the bottom four quintiles, only declining for the top quintile. The authors of the Burkina study comment  that they don’t see any gradients by baseline levels of income either. The Malawi study was not targeted, while the Burkina one targeted to the poor. So, a simplistic take-away message is out of question.
Perhaps a more reasonable approach is to ask where in the distribution of income there is room for improvement, i.e. what is the income/schooling gradient? Looking at the papers from LAC, a continuous gradient is clear. Looking in our data in Malawi, schooling outcomes in the control group don’t improve with baseline welfare for a large portion of the distribution. Even English reading comprehension test scores are pretty constant over the first three quintiles, starting to improve slightly for the fourth and finally jumping for the top quintile. In that kind of a setting, where absolute poverty is higher and the gradient of schooling with respect to income is flatter, it is possible that cash transfer programs will benefit a lot more people than just the bottom decile or quintile. In fact, smaller transfer amounts distributed to a larger number of households may be the better option. One can also see an unconditional cash transfer program targeting itself right out of impact – if the beneficiaries are so poor that the transfer alone is not enough to induce any behavior change.
In economics, we should not look to one study, or even two or more studies, to give us the answer that we can cut and paste into our circumstances. The evidence is complex, settings different, and implementation heterogeneous and completely uncoordinated across studies; all of which to say that all the talk of ‘external validity’ starts to border on nonsensical. I have much more to say about this last point, on which I have been obsessing for weeks, so stay tuned.