Often there are many steps or stages between the starting point of some intervention and its ultimate goal, and at each step, people can drop out. The result can be extremely low power to measure impacts on this end outcome, even though we might be able to detect impacts on the intermediate steps. This post illustrates this point, with the goal of making clear the importance of trying to measure intermediate outcomes, and concludes with suggestions of ways to partially overcome this problem.
Gonzalo was part of a panel with David McKenzie at a recent meeting of the Impact Evaluation Network (IEN). One of the questions during this discussion was whether there were good examples of cases where impact evaluations had found null or negative results, and policymakers had actually changed policy as a result. We thought others would be interested in hearing his examples from Mexico.
It feels like a cold water shower when impact evaluations (IEs) do not show positive impacts. Those studies are neither sexy for academic publication nor for public policy use. But the fact that some IEs show no impact of certain programs or projects, it’s an important piece of information!
I would like to suggest here that if a country has an institutional and relatively credible Monitoring and Evaluation System (M&E), the chances of using IEs with no impact increase.