“If the [NGO] sector wants to properly serve local populations, it needs to improve how it collects evidence.” Donors are also increasingly demanding evidence of impact from NGOs, no longer just the large funders, but the small individual donors as well.
The 'impact study' here seems to be narrowly defined to the WBG flavor of counterfactuals, which work fine if you have a simple intervention, though more complex interventions perhaps demand a more broader and pragmatic view of what is an impact study.
Possible methods for examining the factual (extent to which actual results match what was expected):
Comparative case studies: Did the intervention produce results only in cases when the other necessary elements were in place?
Dose-response: Were there better outcomes for participants who received more of the intervention?
Beneficiary/expert attribution: did participants/key informants believe the intervention had made a difference, and could they provide a plausible explanation of why this was the case?
Predictions: did those participants or sites predicted to achieve the best impacts (because of the quality of implementation and/or favorable context) do so? How can anomalies be explained?
Temporality: did the impacts occur at a time consistent with the theory of change – not before the intervention was implemented?
Possible methods for examining the counterfactual (what would have happened in the absence of the intervention) include:
Difference-in-difference: The before-and-after difference for the group receiving the intervention (where they have not been randomly assigned) is compared to the before-after difference for those who did not. (Difference-in-Differences)
Logically constructed counterfactual: In some cases it is credible to use the baseline as an estimate of the counterfactual. For example, where a water pump has been installed, it might be reasonable to measure the impact by comparing time spent getting water from a distant pump before and after the intervention, as there is no credible reason that the time taken would have decreased without the intervention. Process tracing can support this analysis at each step of the theory of change.
Matched comparisons: Participants (individuals, organizations or communities) are each matched with a nonparticipant on variables that are thought to be relevant. It can be difficult to adequately match on all relevant criteria. (Techniques for improving constructed matched comparison group impact/outcome evaluation designs)
Multiple baselines or rolling baselines: The implementation of an intervention is staggered across time and intervention populations. Analysis looks for a repeated pattern in each community of a change in the measured outcome after the intervention is implemented, along with an absence of substantial fluctuations in the data at other time points.
Propensity scores: this technique statistically creates comparable groups based on an analysis of the factors that influenced people’s propensity to participate in the program – it is particularly useful when participation is voluntary (for example, watching a television show with health promotion messages).
Randomized controlled trial (RCT): Potential participants (or communities, or households) are randomly assigned to receive the intervention or be in a control group (either no intervention or the usual intervention) and the average results of the different groups are compared.
Regression discontinuity: Where an intervention is only available to participants above or below a particular cutoff point (for example, income), this approach compares outcomes of individuals just below the cutoff point with those just above the cutoff point.
Statistically created counterfactual: A statistical model, such as a regression analysis, is used to develop an estimate of what would have happened in the absence of an intervention. This can be used when the intervention is already at scale – for example, an impact evaluation of the privatization of national water supply services.
Possible methods for identifying and ruling out alternative possible explanations
General elimination methodology: possible alternative explanations are identified and then investigated to see if they can be ruled out.
Searching for disconfirming evidence/Following up exceptions
Multiple lines and levels of evidence (MLLE): a wide range of evidence from different sources is reviewed by a panel of credible experts spanning a range of relevant disciplines. The panel identifies consistency with the theory of change while also identifying and explaining exceptions. MLLE reviews the evidence for a causal relationship between an intervention and observed impacts in terms of its strength, consistency, specificity, temporality, coherence with other accepted evidence, plausibility, and analogy with similar interventions.
Contribution analysis: a systematic approach that involves developing a theory of change, mapping existing data identifying challenges to the theory – including gaps in evidence and contested causal links – and iteratively collecting additional evidence to address these.
Collaborative outcomes reporting: this new approach combines contribution analysis and MLLE. It maps existing data against the theory of change and fills in important gaps in the evidence through targeted additional data collection. Then a combination of expert review and community consultation is used to check the evidence’s credibility regarding what impacts have occurred and the extent to which these can be realistically attributed to the intervention.