A week ago we hosted an informal workshop with some academic researchers, policymakers and World Bank staff to review "The second generation of evaluations" of CCT programs. Yesterday's post recounted where we stand with respect to 'Opening the Black Box', parsing out what we can about which design features help to produce the human capital impacts that are one of the programs' twin goals. Today we summarize the rest of the discussion.
A reminder that reducing poverty is often the key objective
Most of the research presented at the workshop focused on the effectiveness of CCTs in achieving one of their twin goals, namely to increase human capital. But as Jere Berhman, Fabio Veras, Norbert Schady and Margaret Grosh pointed out, most CCT programs, especially in LAC, were driven by redistribution as their primary objective. Weak final results on learning or health weaken the belief in 'killing two birds with one stone', but for reasons of equity governments may still want redistributive programs than provide transfers to poor families, even if final human capital outcomes aren't much improved.
In this short run poverty reduction agenda, largish transfers and transfers that increase with family size would be more effective, and their targeting to the poorest important. The workshop was quite short of evidence on this. Few authors presented distributional analysis of their programs. None of the studies brought new evidence to the table on adult work effort. There was new evidence of delays in marriage or fertility for female adolescent beneficiaries for Malawi and Pakistan, but not for total family size among adult women/mothers in the program. Thus the politically powerful fears of labor disincentives and increases in fertility that greatly affect policy design remain, with little new evidence to put them to test or to rest.
CCTs are hoped to reduce poverty in the long run because the children who benefit from CCTs are expected to get more human capital and thus to earn more as adults and escape the intergenerational transmission of poverty. On that front, the evidence to date is thin. Evidence from Brazil shows that Bolsa Familia was successful in delaying the age of entry into the labor market by 1.5 years for boys and in increasing school progression for girls. The long term impact of Oportunidades was shown to be increasing linearly with the duration of exposure to the program, with on average 0.2 additional grades per year into the program. Yet, the long lasting gains on completed schooling do not seem to translate into improved labor market and labor income outcomes: graduates from Oportunidades have not shown improved labor market outcomes at first entry (see here and here). It is possible that it takes time for those gains to materialize. It is also possible that the pathway to long term poverty alleviation might come from gains in health, nutrition or non-cognitive skills. More long term follow-up is needed on these important outcomes.
Chile Solidario, a program that works with socially excluded adults, is one of the second generation programs which combines the short term objective of social assistance with the promotional objective of increased skills and endowments for the current poor. The program was successful in activating the demand for services and the labor supply for those individuals previously detached from the social welfare system, with larger results for women. The Chilean example shows that good employment and earnings depend not just on the individual, but also on active labor market policies tailored to needs of the target population as well as the surrounding labor market.
More hopeful evidence suggests other pathways to longer term poverty reduction. Paul Gertler showed evidence from Mexico that social assistance promotes productive investment through households' investment in durables and investment in micro enterprises. He and co-authors estimated that households save and invest a quarter of their transfers, thereby raising their autonomous income by 22 percent after five years. He also suggests that the investments in housing and durables may open a pathway to better health that hasn't been explored yet in the CCT literature.
How well do CCTs work in lower income settings?
The results so far are mixed and from young and relatively small (Cambodia, Kenya, Indonesia), pilot (Nigeria, Tanzania) or experimental programs (Burkina Faso, Malawi, Pakistan). Results are impressive in Cambodia and Pakistan. Kenya and Indonesia are the first countries where no increase in primary enrollment is observed, and there wasn't much in Nigeria at secondary levels either, where the constraints on the supply side are binding. In Indonesia, primary enrolment rates are already high, payments low and not timed to when school fees are due. In Kenya, there weren't results in primary, but there were in secondary. It seems to us that the lessons from these cases aren't so much about whether CCTs can work in low income settings as they are about truisms in development: that careful diagnostics should precede policy design; that imperfections in policy design or in implementation can preclude impact; that it doesn't make sense to evaluate a program until it is up and running well enough to be given a fair trial. Happily, several of these "plain vanilla" evaluations seem to feeding into policy design and implementations in their respective countries.
New Frontiers
As though there weren't enough on the plate for evaluators and policy makers sorting out the 'old' issues during day one, the discussions of day two wetted our appetite for more. Some highlights:
-
Doing proper cost-effectiveness analysis of CCTs programs is left aside in most evaluations. Jere Berhman led us through some of the difficulties, and how to avoid facile and misleading comparisons, especially for programs like CCTs that have multiple objectives or effects. He made important recommendations (moving towards internal rates of returns, relate benefits to resource costs) that can make the analysis more informative for policy.
-
Behavioral perspectives: Larry Aber made a compelling case for broadening the scope of inquiry and establishing multidisciplinary collaborations with psychologists, sociologists and experts in communication science. Our understanding of the black box needs to be grounded in psychological theories. Self-efficacy beliefs, tradeoffs between intrinsic and extrinsic motivation and child engagement are going to be crucial pathways behind the short and long term gains from CCTs.
-
The critical role of the supply side: CCTs are demand side interventions, and evaluations usually take the supply side as given, or background context. Veronica Silva highlighted the need to set a new research agenda on measurement of access and quality. We need to better understand and document how the supply side responds to the induced new demand for services, as well the set of motivations and incentives and constraints that drive services providers.
-
Governance and political economy affect CCT effectiveness, especially in large scale programs where implementation is done at a very deconcentrated and decentralized level. Claudio Ferraz highlighted that politicians are important actors alongside beneficiary households and service providers: they have electoral incentives to increase program caseload, as well as to minimize the cost of getting rid of beneficiaries that do not comply with the conditionalities. He also warned about the risk of political gridlock that may arise from large and established redistributory programs.
- CCTs in the broad context of social policy and social protection: Helena Ribe, Antonio Claret and Agustin Escobar reminded us pioneering CCT programs revolutionized the delivery of social assistance making it more systematic, better targeted, more accountable; Now countries with mature programs are moving on in their social protection agendas, especially considering layering on services, or making linkages among them. However, evaluations of Chile Solidario and Guanajuato (in Mexico) state's Contigo Vamos Por Mas program showed how hard that can be – both in terms of their implementation as well as their evaluation. The research community should expand evaluation methods that evaluate comprehensive strategies and systems beyond the focus on single programs.
We are grateful to the Spanish Impact Evaluation Fund for the support of the workshop and of six of the evaluations presented.
Join the Conversation