When state institutions find it a challenge to deliver services in under-resourced areas, its common for policy makers to consider leveraging existing local non-state capacity to help. This involvement of NGOs or CBOs is meant to supplement the state as service provider but a recent paper by Ashis Das, Eeshani Kandpal, and me demonstrates possible pitfalls with this extension approach. Just as implementation capacity of governments is a key determinant of government program performance, NGO capacity is a key determinant of NGO performance and under-resourced areas are likely to contain under-resourced local organizations. We find this to be the case in our study context of malaria control in endemic regions of India. Besides highlighting this challenge, our results also highlight the difficulties that small-scale evaluations present to the generalizability of findings, especially those implemented by non-state actors. Implementation capacity can be a key confounder of generalizability and it is not often measured or even discussed the current practice of impact evaluation needs to think harder about measures that capture implementation capacity in order to generalize IE results to other contexts.
Our study examines the use of local NGOs to supplement malaria service delivery in the Indian state of Odisha. We find that, indeed, the inclusion of NGOs can enhance services and improve outcomes a good thing given the high disease burden in the endemic districts where the study took place but these supportive services did not translate into improved outcomes for all areas considered.
First, what exactly was evaluated? In two endemic districts in India, NGOs were contracted and trained to provide community mobilization activities to promote consistent malaria bed net usage as well as preferred care seeking behavior for fever local community health workers (known as ASHA) had been trained in the latest diagnostic and curative technologies for malaria and the Indian malaria control program wished to encourage a switch in fever care seeking from traditional providers to ASHA. So the NGOs used several forms of local media to promote these official messages. Further, in a subset of villages the NGOs also provided supportive supervision directly to ASHA in the form of enhanced training, problem solving, and encouragement.
Extensive consultation with local health officials and community leaders identified exactly three NGOs that fulfilled the stipulated criteria for implementers to have (a) previous experience in malaria-related activities and (b) previous activity in the selected areas of study. (It is important to note that there was little choice of potential implementers, and all possible implementers were included in the study.) The three NGOs, one in District A and two in District B, were contracted, trained, and assigned non-overlapping areas in which to operate. This was a standard three arm RCT mobilization plus supportive supervision, mobilization alone, and a control with randomized exposure across study villages.
It turns out that the effect of NGO involvement on net usage and fever care-seeking patterns the main targeted outcomes of interest varies significantly by the district of implementation. Reported net usage is already fairly high in both districts, with 86 percent of household members in control villages sleeping under a net in District A, and 73 percent in District B. Despite the higher rate of control village usage of nets in District A, an even greater fraction of the treated villages was reported to sleep under mosquito nets after the mobilization and supervision efforts. This was not the case in District B.
The results also indicate that the intervention, especially the arm with supportive ASHA supervision, was successful in prompting fever patients to switch care seeking from an unskilled provider to the ASHA. Yet again its the results for District A that drive the overall findings. In control villages in District A, 17 percent of all sampled fever patients saw an ASHA worker. However this rate is 32 to 36 percent in the intervention villages, depending on the study arm. The degree of fever care seeking change is far less in District B, and none of the impacts are statistically significant in that district.
This difference in intervention effectiveness across districts can be due to several (possibly overlapping) reasons including (a) differential population and area characteristics, (b) differential health worker characteristics, and (c) differential implementation. So we next investigate each possibility in turn.
There are indeed some systematic differences in population characteristics between the two districts with, for example, District A containing a smaller proportion of scheduled caste households and a higher proportion of Hindu households. It turns out there are no essential differences in health worker characteristics across districts, but there are indeed a few salient observable differences in NGO characteristics. The District B NGOs had equivalent staffing and training numbers to the District A NGO but they had little previous experience in the key activities of community outreach and demand generation, while the District A NGO had several years of experience. Despite the lowered effectiveness, the District B NGOs actually spent more implementation funds per village.
So we find that differences in observed outcomes can mostly be ascribed to differential population and implementer characteristics, as well as their possible interaction. We can press further with decomposition methods (I've previously blogged about these methods here) that seek to balance observed characteristics through regression decomposition or propensity weighting. When we do this we find that, for the key indicators related to prompt fever care seeking, the cross-district differences in program effectiveness persist after we control for all observable population differences. While we cant rule out the possibility that differential program effectiveness is solely driven by some unobserved population characteristic (even though we control for a rich set of observables), these results are consistent with the proposition that implementation differences plays a key role in differential program performance even when the implementers are all local NGOs.
In this sense our findings are similar in spirit to the recent paper by Bold and co-authors that demonstrates differential effectiveness depending on whether a Kenyan education program is implemented by a government agency or an NGO. They find that NGO implementation has a significant and positive effect on students test scores but government implementation results in no change. Here we find that even among small local non-state actors, program effectiveness diverges significantly in some settings implementation effectiveness is not just a story of government vs. NGO.
So our take-away messages are two-fold:
1. In low-resource areas with few high quality local institutions, there may be little alternative to intensified state investment or state-led incentives to improve service, at least in the near term. The lack of local non-state capacity in many under-served and under-resourced areas speaks to the complex interactions between poverty and local characteristics. Quick fixes through the utilization of non-state capacity may be difficult to attain.
2. Small scale IEs that hope to inform policy need to consider the capacity of the implementer, as well as the characteristics of the population and setting, and need to understand how these characteristics, both individually and jointly, mediate program impacts. Without standardized measures of implementation capacity, it will be difficult to generalize policy lessons from any small-scale evaluation.
Join the Conversation