We hear a lot about how pilot interventions, run by an NGO, have a big impact in randomized control trials. The recommendation of the study is to scale this up. But policy skeptics say: “oh no, that’s an NGO pilot, it won’t work if government does this as scale.” Into this debate comes a fascinating new paper by Faraz Usmani, Marc Jeuland, and Subhrendu Pattanayak.
Usmani and co. look at how prior exposure to an NGO affects take up and use of a clean-cookstove intervention. Bottom line: it matters a lot. What may be surprising is that this happens when the research team, not the NGO delivers the actual intervention. So it’s about prior exposure and experience – not NGO delivery experience or familiarity with context and clients.
Some background. The setting is 97 hamlets in 38 villages in the Indian state of Uttarakhand, where more than half of the households are below the poverty line. Folks are having issues with their cooking technology, with more than half believing that the smoke from their current stove is unsafe, and about 20 percent reporting a cough or cold in the household prior to the baseline. Usmani and co. send out a trained team of enumerators, who identify as being affiliated with an (environment and development) NGO that has worked in some of these communities before. The team gives households information, provide demonstrations of an improved biomass stove and an electric stove, and then offer the household the chance to buy either or both of these stoves on an installment plan with a 2-30 percent price rebate (randomized).
Now this is a tricky experiment to set up. NGOs don’t usually randomly visit communities. So Usmani and co. take data on villages where the NGO had worked in and then do propensity score matching with census data to find nearby similar non-NGO villages. They walk us through their matching approach and take a conservative approach (e.g. trimming) to get a decent comparator sample. But that’s not all, they also go in in detail, looking at contextual factors (e.g. presence of organized groups within communities) and winnow it down further. In the end, they have 38 villages: 19 NGO and 19 non-NGO. Within these villages, they randomly select between two and four hamlets to end up with a total of 97 hamlets. They then roll out the improved cookstove (ICS) intervention to households in 70 hamlets (remember, they’re randomizing the rebate here – and it’s across three levels).
Usmani and co. have a baseline, mid-line and endline. The three rounds let them use a household fixed effect and look at prior experience with the NGO at each follow-up round. This helps them deal with some of the potential unobservable characteristics (if you are worried after the matching).
The results are interesting. First, a lot of the households take up the cookstoves. And, right off the bat, there is a marked difference: take up is 45 percent in non-NGO households and close to 60 percent in NGO households (this difference is significant). And those exposed to the NGO before also keep and use their stoves – at midline they come in around 16 percentage points higher on these measures. Interestingly, while the NGO exposure effect remains positive at endline, it’s no longer significant (although it’s often not significantly different from the midline effect). Usmani and co. point out that this could be due to breakage – about 15 percent of all users reported problems with their stoves.
Usmani and co. also show that NGO-exposed households report significantly less fuelwood use at midline (again, attenuating at endline) and also report significantly less time collecting fuel at both midline and endline.
So these are fascinating results: the prior exposure to an NGO really increased uptake and use of these cookstoves. With attendant welfare gains. (it’s worth noting that the non-NGO villages didn’t do horribly either, but they definitely didn’t do as well). The big question is why? Usmani and co. develop a model and argue that this is due to the NGO lowering transaction costs to technology adoption. Unfortunately, they don’t have the data to nail down this versus other explanations. So, this is an area for fruitful additional research.
Indeed, this is one of those papers that fires up the next-question generator. Would this work for (capable) governments not just NGOs? And, if so, how capable? Is it increasing in how capable the institution is? Does it matter how close the new intervention is to the prior experiences with the service provider? Is this a technology adoption specific effect (the results on usage seem to suggest it’s not just this but…)? So much more to do. Exciting!
Join the Conversation