In a recent entry to this blog, Gabriel Demombynes presents an interesting discussion about the role of the implementer of interventions (for instance, an NGO versus the government) and the possibility to replicate the intervention. He points out that we seem to be able to replicate interventions by NGOs, but find it more difficult to replicate government led interventions. He forcefully argues that we, nonetheless, should work more with governments, but with interventions that vary the modality of service delivery. In this blog, I want to present another perspective of the problem: regardless of the implementer, supply driven interventions—which have proven to be difficult to replicate across different contexts and countries—are institution-dependent interventions. In contrast, demand driven interventions—which we have been able to replicate in different context and countries—depend on the behavior of one institution (the family) for which we have better behavioral theory. If this hypothesis is true, then the approach of development economics that calls for the gathering of strong, sound evidence from (usually) randomized-controlled trials on the causal impact of different interventions in different countries needs to go deeper than just replicating an intervention by exploring what works and why.
Demand-side versus supply-side interventions. There are some demand-side policies that have yielded positive results in several countries. For example, conditional cash transfers (CCT), in which (usually poor) families receive cash in exchange for certain behaviors—for instance, keeping children in school—have been shown to systematically increase enrollment and attendance rates (in some cases, in a very significant way). The popularity of these interventions followed the success of Progresa / Oportunidades in Mexico. After Progresa was found to be effective, very quickly, several other developing countries followed suit and the evaluations of these programs show similar effects. Other interventions, such as programs that reduce user fees or programs that provide information to students about the rate of return to education show consistency in results on enrollment and attendance across different countries (see the review by Fiszbein and Schady here).
In contrast, we have not been so successful in replicating the results of several supply-side interventions. We have found inconsistent results from interventions like teacher incentive programs, computers for education, and programs providing additional inputs to schools, among others. For example, take the case of decentralizing to parents the decision making power regarding schools (or school based management programs, SBM). The idea behind SBM is that parents have better information on the needs and the operation of schools than the central government; parents are the direct clients and the most interested in guaranteeing that schools are of good quality; and parents can exercise direct accountability over the school. SBM programs give school boards the power and resources to take decisions. Three rigorously evaluated interventions —two in India (papers here and here) and one in Kenya— yield very different results. While both evaluations in India show that the programs were actually implemented (that is, parents were more aware of their potential power and they knew better their functions and rights), the evaluations show no effects on educational outcomes. However, the similar intervention in Kenya shows positive effects. The authors of this paper argue that the intervention was successful because it induced higher effort from a contract teacher (without a secured contract and strong incentives), and it did not impact regular teachers (with stable contract and weak incentives). Recent systematic reviews (here and here) points towards these discrepancies in results.
It can be argued that inconsistency in results emerges from inconsistency in the implementation of the policies in different contexts. In one country the intervention may have been implemented successfully, whilst in others not. Clearly, this explanation is plausible; however, I believe that there are some indicators that do not, in general, support the hypothesis. In the majority of the program evaluations with inconsistent results, the evaluations do show that the program was indeed implemented. For example, in teacher incentive programs, the teachers know about the program; in school based management programs, the parents know their roles and rights after the implementation; in programs providing computers for schools, the computers actually arrive to schools, and teachers receive training; and in textbooks for schools, the schools receive the textbooks and teachers use them more than in control schools. Something beyond the lack of implementation differs in the different contexts to produce inconsistent results. Furthermore, CCTs have been implemented in with varying strategies and dissimilar context, but nonetheless most of the programs have shown positive effects.
Institution dependency and supply-side interventions. If the explanation is not in the implementation, what can explain these results? The explanation may emerge from the asymmetry between demand-side interventions which depends on household behavior, and supply-side interventions which depends on institutional behavior.
A large array of demand-side interventions transpire from the idea that opportunity cost is an important determinant in decisions regarding investment in education and that families balance the costs and the benefits from education. Lower educational costs —for example, from conditional cash transfers— induce a more favorable benefit-cost ratio, and therefore, the probability of school enrollment increases. These results are based on solid theoretical models on household behavior.
In contrast, supply-side interventions depend on institutions that vary greatly across countries. A program of SBM will have certain results under certain types of institutions, and not in others. For example, when the figure of the teacher is culturally very strong, SBM programs that give power to parents may not be effective because the parents will not demand certain actions from the teacher. However in other contexts, such as is hypothesized in the case of Kenya, the conjunction of SBM and hiring a contract teacher (subject to contract renewal based on performance) may induce positive results because parents will have the power to exercise action, and the institution (the different contract for the extra teacher) may elicits changes in teacher behavior. In sum, supply-side interventions heavily depend on institutions that dramatically differ from context to context, whereas demand-side interventions depend on a more homogenous institution that in principle does not differ greatly across contexts. If this is the case, the replication of supply-side interventions in different context is set to fail since they are institution-dependent, and consistent results will be difficult to obtain without limiting the replications to situations that are virtually identical. Clearly, this problem goes beyond external validity.
The implication of this potential hypothesis is that the focus of the next generation of evaluations should be not so much on what works (and the channels, as some people have suggested) but on the types of institutions that make the policy work. By necessity we need to start thinking what are the relevant institutions, and their interactions with policies. If the intervention is institution-dependent, we can learn something important for the same context in which we are making the evaluation (for instance, scaling up a pilot program citywide), but we cannot learn “what works” by looking across countries. Either we can start exploring the relationship between institutions and interventions, or we need to rebuild each supply-side program in each new context. In this sense I believe that the problem of institution-dependent policies is a sobering –and more challenging—one.
Felipe Barrera-Osorio, is an Assistant Professor in the Graduate School of Education at Harvard University. His faculty homepage is here.
Join the Conversation