I was recently working with an implementing agency to design an impact evaluation and we were having trouble reaching a point where there was going to be a viable impact evaluation that answered big questions about the efficacy of the intervention. Looking back, part of the problem was that this agency was the implementer, not the funder. And they were paid by the funder based on reaching a certain number of people and having those people participate in the program.
From a funders point of view this makes a lot of sense. For a long time, a lot of aid was measured/judged based on inputs: x dollars disbursed, y school books purchased, and the like. A number of donors have started to move away from this -- for example, the World Bank has a lending instrument which disburses against results as implemented by the recipient country. In the grant-based aid realm, other donors directly contract implementing partners (sometimes for-profit firms, sometimes NGOs) to run things. And they write these contracts (as a smart person would) based on actually delivering a program and getting people to participate. Finally, a number of governments write contracts like this with their own ministries (or private providers) and then hold the relevant officials to account when performance targets aren't met.
Now, as you've probably picked up by now, most of these programs' definition of results or performance stop with outcomes (i.e. we offered a program, people used it). They don't use impacts because impacts may take awhile to manifest and, of course, because a zillion things are confounding the impacts. This makes sense from a contracting point of view.
The problems arise if we are trying to learn something, in a rigorous (i.e. impact evaluation) fashion. The first problem is that this, in some cases, can discourage innovation and risk taking in programs. If my contract says that I have to reach 10,000 firms with a certain set of information and I have a way to reach the firms that has worked in the past, I'll go with that, even if there are some innovative but unproven potential cheaper ways.
The second problem is that if the contract pays the implementer for "performance" then the deck is stacked against a rigorous evaluation. Impact evaluations are rarely costless for the implementer; things such as needing to recruit or identify a control group, implementation delays due to survey rounds, and the like can add to the cost of program implementation (I discussed these costs in more detail in an earlier post). And if the implementer has to reach a target they would prefer to put all of their cash towards that. Or, if they're a for-profit firm, they definitely won't want to cut into those profits to produce the public good that is impact evaluation.
The obvious thing to do is to put the impact evaluation in the contract. That is, even if the evaluation will be done by a third party, specify that the implementer has to cooperate with the evaluation (and bear the costs). This isn't as easy as it sounds, not least because the parameters of the impact evaluation are often going to be realized after the preparatory implementation work has been done (which might be too late to alter the parameters of the implementation contract). Despite this, I have seen some of the folks who manage these implementation contracts do this effectively. But a) they are the early adopters, the innovators, and b) they are a minority. As this mode of programming becomes more common, my concern is that we will increasingly neglect to include this. And, as I found out during this recent trip, trying to stick an evaluation in a contract that is already written is woefully hard, if not impossible.
It would be good to hear from others on this -- have you had experiences with this? Any advice on designing the program implementation contract? Other thoughts?
From a funders point of view this makes a lot of sense. For a long time, a lot of aid was measured/judged based on inputs: x dollars disbursed, y school books purchased, and the like. A number of donors have started to move away from this -- for example, the World Bank has a lending instrument which disburses against results as implemented by the recipient country. In the grant-based aid realm, other donors directly contract implementing partners (sometimes for-profit firms, sometimes NGOs) to run things. And they write these contracts (as a smart person would) based on actually delivering a program and getting people to participate. Finally, a number of governments write contracts like this with their own ministries (or private providers) and then hold the relevant officials to account when performance targets aren't met.
Now, as you've probably picked up by now, most of these programs' definition of results or performance stop with outcomes (i.e. we offered a program, people used it). They don't use impacts because impacts may take awhile to manifest and, of course, because a zillion things are confounding the impacts. This makes sense from a contracting point of view.
The problems arise if we are trying to learn something, in a rigorous (i.e. impact evaluation) fashion. The first problem is that this, in some cases, can discourage innovation and risk taking in programs. If my contract says that I have to reach 10,000 firms with a certain set of information and I have a way to reach the firms that has worked in the past, I'll go with that, even if there are some innovative but unproven potential cheaper ways.
The second problem is that if the contract pays the implementer for "performance" then the deck is stacked against a rigorous evaluation. Impact evaluations are rarely costless for the implementer; things such as needing to recruit or identify a control group, implementation delays due to survey rounds, and the like can add to the cost of program implementation (I discussed these costs in more detail in an earlier post). And if the implementer has to reach a target they would prefer to put all of their cash towards that. Or, if they're a for-profit firm, they definitely won't want to cut into those profits to produce the public good that is impact evaluation.
The obvious thing to do is to put the impact evaluation in the contract. That is, even if the evaluation will be done by a third party, specify that the implementer has to cooperate with the evaluation (and bear the costs). This isn't as easy as it sounds, not least because the parameters of the impact evaluation are often going to be realized after the preparatory implementation work has been done (which might be too late to alter the parameters of the implementation contract). Despite this, I have seen some of the folks who manage these implementation contracts do this effectively. But a) they are the early adopters, the innovators, and b) they are a minority. As this mode of programming becomes more common, my concern is that we will increasingly neglect to include this. And, as I found out during this recent trip, trying to stick an evaluation in a contract that is already written is woefully hard, if not impossible.
It would be good to hear from others on this -- have you had experiences with this? Any advice on designing the program implementation contract? Other thoughts?
Join the Conversation