A week ago we hosted an informal workshop with some academic researchers, policymakers and World Bank staff to review "The second generation of evaluations" of CCT programs. We finally have the website, where you can see all the presentations made available by the authors and video of the event. Two posts in the Development Impact blog (here and here) go into more detail on the effectiveness of conditions — their theory, evidence and the conflicting values around them. In blog posts today and tomorrow we’ll summarize discussion of the whole workshop. Today we introduce it and focus on the human capital formation side of things. Tomorrow we consider the poverty reduction objective, how CCTS are working in low income countries, and some "new frontiers" with respect to considering behavior, governance, supply, costs, and wider social protection strategies.
On Day 1 we rampaged through 17 evaluations from 14 countries, all done since the Policy Research Report on CCTs was published in 2009. We saw evaluations of huge national programs and tiny comparative treatment arm experiments and debated the pros and cons of impeccable control groups versus questions about external validity. On day 2, we paused to reflect on the new frontiers of CCTs, when several more evaluations were interjected into the discussion. For a change the big majority of evaluations were from countries outside of Latin America — 5 from Africa, 1 from the Middle East, 2 from East Asia and 1 from South Asia.
The clearest and most unanimously agreed message from the workshop is that there is an increasingly long vector of outcomes shown to be positively affected somewhere by some CCT program — many concerned with health and education, but also with work, livelihoods and investments in productive or housing investments, and another vector associated with youth risky behaviors and mental health. This should not perhaps be surprising, as we know that poverty brings with it a dizzying array of deprivations and risks. But it has implications for thinking about the role and design of transfer programs, and for ensuring that when evaluations are done, they are comprehensive. We may have been underestimating total benefits or not captured unintended consequences because we weren’t measuring them all.
Opening the Black Box — Which Design Features Matter in Producing Human Capital Outcomes?
We improved our understanding of the black box, but there are still not very firm answers on some key questions, partly because the findings vary, and partly because CCTs involve multiple outcomes and different countries (or analysts) give them different values.
Do Conditions Matter?
The role of conditionalities took a central stage at the opening of the workshop. The issue is important, as conditions can have big implications for who will and won't benefit from the transfers, may impose costs on households, and monitoring their compliance requires some serious administrative systems. These tradeoffs are especially critical in low-income settings. The first careful and innovative study going at the heart of the question is the Malawi scholarship experiment for adolescent girls. As Berk Ozler pointed out in his presentation (and here) conditional transfers were more effective than unconditional transfers in raising school enrollment and attendance, which is the outcome made salient by the conditionality. Yet, unconditional transfers brought about improvements on a larger set of outcomes, ranging from the choice of less risky sexual partners, to delayed fertility and marriage. In Burkina Faso, conditions did matter to enrollment of younger children (late school start is common in Burkina) and for the use of preventive services. In the Morocco multi-arm experiment, the conditionality did not really make a difference, possibly because many households did not understand how it was supposed to work.
The puzzle of the low elasticity to the transfer amount
The evidence from four scholarship programs with comparative treatment arms (in Cambodia, Morocco, Malawi and Nigeria) and from a pilot in Pakistan seems to indicate that the small benefit amounts, on the order of 3-5% of household welfare, yield large increases in enrolment.
The implications of these findings are potentially powerful, but we aren't sure the question of how much to pay is fully settled.
First, these results stand in contrast with evidence from Latin America where the direct/opportunity cost of schooling may be binding. New research from Progresa/Oportunidades shows that the program raises secondary school enrolment only for those households whose family network includes households with primary school children. As Francisco Ferreira pointed out in his discussion, we should learn more about the range of transfers amounts where the elasticity is still high, and that may vary by context and program objectives.
Moreover, though the potential to use small(er) transfers to produce the desired behavioral changes in enrolment, retention or graduation, is undeniably attractive, raising human capital is often NOT the main or only objective of CCTs, and low transfers are much less effective for the short run poverty objectives of these programs.
The power of framing of conditions
The idea that it is neither cash nor conditions that drive results, but "something else" got a lot of discussion. Ben Olken suggested that behavior will depend on the way the message of the conditionality is framed and internalized by the households. Ester Duflo suggested that in the Morocco experiment, the increases in schooling resulting from a relatively small cash transfer not understood to be conditional are likely to be driven by the increased awareness and salience of education. The qualitative work from Malawi also emphasized the importance of the 'culture of school' as a key factor in explaining the gains. How expectations, aspirations and projections into the future get changed by these programs is still an unexplored but critically important area of research in the future. Larry Aber convincingly argued for the need of a theory of change, where mechanisms, mediators and moderators are clearly outlined, as well as and for the need to broaden the scope of inquiry, fostering interdisciplinary work with psychology and communication science. Economists are now making the same call for 'mechanisms' experiments to complement policy evaluations and to directly test those parts of the causal chain (mediators) where the knowledge gap is largest (see here).
The new nuanced evidence on the intra-household decision making:
The experiments for Malawi and Opportunity NYC helped us learn that on the surface there is no conflict of interest between the parents and the children in terms of educational outcomes. However, enrollment and school attendance aren’t the only outcomes that matter. In Malawi, the mental health of the girl deteriorates where large transfers to her parents depend on her school attendance. In contrast, in Opportunity NYC, there was no decline at all in mental health among beneficiary children. On the other dimension of intra-household decision making, the Burkina Faso and Moroccan experiments imply that issuing payments to mothers rather fauthres seems to be marginally more effective for education outcomes, but not for health. Indeed, Richard Akresh ended his presentation on the Burkina Faso experiment with the words "fathers aren't so bad". The explanation for the different results is likely to lie in how income pooling and separate spheres of command vary across cultures, which begs for more situation specific and qualitative work to really understand the underlying pathways.
Going from conditions on use of services to those on outcomes
As Norbert Schady suggested in his final remarks: maybe "you get what you pay for" that is, since CCTs are targeted on enrolment or attendance, we should not expect impacts on learning. After all, prior evidence showed no additional learning despite increased schooling in Mexico, Colombia, and Cambodia. New evidence presented in the workshop for Morocco, Nigeria and Malawi was more optimistic, showing learning gains in all three countries.
We need to understand more fully the interplay of different factors in different settings. Lack of final outcomes could be due to the fact that the program brought or kept less advantaged children in school (selection effect) or due to the lack of response of the supply side (crowding). Perhaps, if maximizing learning is the main objective, this could be achieve by either incentivizing graduation and tertiary enrolment (as in a Colombian experiment analyzed by Barrera-Osorio et al 2011), or explicitly targeting test scores (as in a Mexican experiment Berhman, Parker, Todd, Wolpin 2011), where providing incentives to both the demand (students) and the supply side (teachers) yields larger gains in tests score than giving incentivize to each separately. In a broader human capital policy perspective, Costas Meghir pointed out that investment in early childhood development would increase the returns to later schooling investments. While transfers may play an important role in this domain (see here and here), it is not at all clear whether conditionalities beyond those already present for health or health education are needed or desirable.
And of course equity and efficiency need to be balanced. If poorer students are less academically prepared, basing conditions on performance may make it difficult for the poor to comply and benefit, a consideration which has been given heavy weight by Latin American policymakers.
Stay tuned for the rest of the story tomorrow.
We are grateful to the Spanish Impact Evaluation Fund for the support of the workshop and of six of the evaluations presented.
Join the Conversation