Sometimes (maybe too many times), I come across an evaluation with middling or null results accompanied by a disclaimer that implementation didn’t go as planned and that results should be interpreted in that light. What can we learn from these evaluations? Would results have been better had implementation gone well? Or even if implementation had gone just fine, was the intervention the right solution for the problem? It’s hard to say, if we think of program success has a product of both implementation and a program that is right for the problem. Zero (implementation) times zero (appropriateness of the program) equals zero and zero times 300 also equals zero (for a more eloquent and entertaining version of this, see Dean Karlan’s lecture here). What’s even more frustrating is that sometimes we spend hundreds of thousands of dollars on an experiment to learn that something only partially implemented doesn’t work.
This is why the Strategic Impact Evaluation Fund just put out a call for proposals focused on what we’re calling nimble evaluations, or rapid, low-cost evaluations. We would like these evaluations to generate experimental evidence on implementation, for them to be a space where economists and other social scientists can tinker like plumbers or set up mechanism experiments so that that longer, more expensive evaluations focused on outcomes can happen for interventions that have a high chance of being implemented. To fix ideas, think of an early childhood development program. We will obviously want to know if this program improves child development or even earnings or other markers of success later in life. But that would be a very long and very expensive evaluation, particularly because measuring child development well requires substantial investment. And it’s not even clear if we should do this if we don’t have a clear idea of how to deliver the program in the first place, if we think take-up, adherence, or the quality of service delivery are likely to be serious challenges. A nimbler evaluation could first test who is more likely to deliver the intervention on time, volunteers minimally incentivized with something like a T-shirt or small stipend (current default in many programs), or individuals who are regularly remunerated for their work.
With this call for proposals, we’re experimenting with two new features for supporting experiments and quasi-experiments in low- and middle-income countries. First, researchers will have the opportunity to design an evaluation for the nearly 50 World Bank or DFID projects that have expressed interest in building in nimble evaluations into project design to address implementation challenges. For example, we have project trying to improve immunization take-up in Pakistan, and we have a project in Djibouti trying to ensure the delivery and take-up of a package of early childhood development interventions linked to a conditional cash transfer. In addition to the satisfaction of helping a government refine a program or policy, researchers designing evaluations for these projects need not separately fundraise for implementation funding as this would be covered by the projects. What’s more is that our project teams will do their best to facilitate access to administrative data. Who doesn’t get excited by health insurance claims data or daily attendance data on students?
Second, with this call for proposals and the next one, we’re trying to encourage iterative experimentation. For more than a decade, everyone has recognized the importance of this (for example, Banerjee and Duflo, 2009; World Development Report 2015; World Development Report 2018; and Andrews, Pritchett, and Woolcock, 2017), but from a funder’s perspective, this is not easy. It would involve funding an engagement with a government or NGO, not a single evaluation, with undefined objectives, uncertain outputs, and unobserved chemistry between researchers and implementers upfront. To mitigate this risk but still provide incentives for iterative experimentation, SIEF will open a follow-up call for proposals, intended for evaluation designs that build upon previous results and open only to teams that receive funding and complete an evaluation during this current round.
Of course, we’ve chosen particular ways to deal with the challenges of sparse experimental evidence on implementation and underinvestment in iterative experimentation, and perhaps we’ll have to tinker with these in an iterative way as we see how these next two calls for proposals pan out. In the meantime, we hope all you brilliant researchers out there will apply to the nimble call and create much needed evidence on how to implement interventions to build human capital in low and middle income countries.
This is why the Strategic Impact Evaluation Fund just put out a call for proposals focused on what we’re calling nimble evaluations, or rapid, low-cost evaluations. We would like these evaluations to generate experimental evidence on implementation, for them to be a space where economists and other social scientists can tinker like plumbers or set up mechanism experiments so that that longer, more expensive evaluations focused on outcomes can happen for interventions that have a high chance of being implemented. To fix ideas, think of an early childhood development program. We will obviously want to know if this program improves child development or even earnings or other markers of success later in life. But that would be a very long and very expensive evaluation, particularly because measuring child development well requires substantial investment. And it’s not even clear if we should do this if we don’t have a clear idea of how to deliver the program in the first place, if we think take-up, adherence, or the quality of service delivery are likely to be serious challenges. A nimbler evaluation could first test who is more likely to deliver the intervention on time, volunteers minimally incentivized with something like a T-shirt or small stipend (current default in many programs), or individuals who are regularly remunerated for their work.
With this call for proposals, we’re experimenting with two new features for supporting experiments and quasi-experiments in low- and middle-income countries. First, researchers will have the opportunity to design an evaluation for the nearly 50 World Bank or DFID projects that have expressed interest in building in nimble evaluations into project design to address implementation challenges. For example, we have project trying to improve immunization take-up in Pakistan, and we have a project in Djibouti trying to ensure the delivery and take-up of a package of early childhood development interventions linked to a conditional cash transfer. In addition to the satisfaction of helping a government refine a program or policy, researchers designing evaluations for these projects need not separately fundraise for implementation funding as this would be covered by the projects. What’s more is that our project teams will do their best to facilitate access to administrative data. Who doesn’t get excited by health insurance claims data or daily attendance data on students?
Second, with this call for proposals and the next one, we’re trying to encourage iterative experimentation. For more than a decade, everyone has recognized the importance of this (for example, Banerjee and Duflo, 2009; World Development Report 2015; World Development Report 2018; and Andrews, Pritchett, and Woolcock, 2017), but from a funder’s perspective, this is not easy. It would involve funding an engagement with a government or NGO, not a single evaluation, with undefined objectives, uncertain outputs, and unobserved chemistry between researchers and implementers upfront. To mitigate this risk but still provide incentives for iterative experimentation, SIEF will open a follow-up call for proposals, intended for evaluation designs that build upon previous results and open only to teams that receive funding and complete an evaluation during this current round.
Of course, we’ve chosen particular ways to deal with the challenges of sparse experimental evidence on implementation and underinvestment in iterative experimentation, and perhaps we’ll have to tinker with these in an iterative way as we see how these next two calls for proposals pan out. In the meantime, we hope all you brilliant researchers out there will apply to the nimble call and create much needed evidence on how to implement interventions to build human capital in low and middle income countries.
Join the Conversation