Syndicate content

Millennium Villages Project continues to systematically overstate its effects

Gabriel Demombynes's picture

by Michael Clemens and Gabriel Demombynes

The Millennium Villages Project (MVP) is an experimental anti-poverty interventionin villages across Africa. In October, we released evidence that the Project’s official publications were overstating its real effects, and we offered suggestions on improving its impact evaluation. On Tuesday the MVP, whose leadership and staff are aware of our work, continued to greatly overstate its impact.

It started last year. In a June 2010 report called Harvests of Development, the Project claimed that the impacts of the project included expanded cell phone ownership.  For example, the MVP claimed that increases in cell phone ownership at the Ghana site were caused by the project, in this extract from page 91 of the MVP report:

This claim has little basis, because cell phone ownership has been expanding at about the same rate all around the MVP site in areas untouched by the project. The graph below, from our paper, shows cell phone ownership at the MVP site in black compared with various other areas:

No reasonable person could look at these findings and conclude that the MVP intervention clearly caused any substantial increase in the rate of expansion of cell phone ownership. The evidence gives no clear reason to believe that cell phone ownership would have expanded any more slowly at the intervention sites if the Project had never existed.

But on Tuesday, months after multiple discussions we’ve had with MVP leaders on our research, a post on the MVP’s blog restated the claim that the increase in mobile phone ownership at the intervention sites was caused by the Project, calling the increase at the Sauri, Kenya site one of the MVP’s “achievements.”

As the World Bank Chief Economist for Africa Shanta Devarajan has observed, our evidence does suggest that the MVP has had some positive short-run impacts on people’s lives. So there is no reason to overstate the impacts.

Before-vs.-after comparisons implicitly assume that in the absence of the intervention, nothing would have changed. This assumption is demonstrably incorrect in the case of the MVP and cell phone ownership. The cell phones are just one example of how the Project overstates its impacts; in the paper we discuss others. For a brief summary and a discussion of my visit to the Sauri MVP site, see this earlier post.

Comments

Submitted by Anonymous on
Thank you for this. While I applauded the concept of the MVP in principal, my experience trying to collaborate with them in the field is that their ego and desire to treat these villages as their little closed off labs (and not communities people actually live in) makes them hard to work with. I'm sad to see this kind of "impact evaluation" but appreciate the reality-check you provide here.

Submitted by olugbenga adesanya on
Until the project is peopled centered and not built around arrogant government officials and legislators, MVP shall address the effect and not the cause. Nigeria is not too good and example to emulate. Cell phone has little bearing with development as there are millions of cell phones that compete with food and other necessities in the spending plan of the Nigerian citizen.

Submitted by Jacob AG on
...we don't know whether development in the MVs has had any effect in the surrounding areas. So, while you are correct to say that the simple before-and-after evaluations of the MVP are not sufficient to draw causal inferences about the project's success, neither are your recommended comparisons between MV sites and the surrounding areas. What would cell phone ownership in the surrounding (non-MV) areas have looked like without the MVP, for example? Maybe they would be the same, or maybe they would be worse. We don't know, and I'm not sure that we ever will. Ultimately there are limits to what we can know about the MVP's success (or lack thereof). Good work, by the way. I'm glad that you two wrote that paper, and that you continue to follow up on the deficiencies of the MVP's evaluation.

Jacob, Thanks for the perceptive comment. We are careful to say that in the paper that our differences-in-differences estimates are not rigorous impact evaluation estimates. Our paper argues for a rigorous impact evaluation design that the MVP could have implemented (and could still implement as it goes forward creating more sites.) To address your point directly: the Sauri MVP site constitutes 1.3% of the population of rural Nyanza, which we take as the first point of comparison for the changes in Sauri. While it is possible that there were some spillovers to neighboring areas, it is extremely unlikely that interventions at Sauri had measurable effects for rural Nyanza *as a whole*. We also compare what happened to Sauri with rural Kenya overall and all Kenya. The rates of growth of cell phone ownership are identical in all three areas. It is implausible that anything that happened in Sauri had any affect on cell phone ownership rates in the country as a whole. Even more broadly, there has been an explosion in cell phone ownership across sub-Saharan Africa, including in countries with no MVP sites. Collectively, this provides fairly strong evidence that the MVP has had no effect on cell phone ownership rates.

Submitted by olugbenga adesanya on
Cell phones are more of status symbol and not evidence of growth.

Submitted by Raj Raina on
Thank you for your active discussion on the topic. A question on a quote from a response post dated oct 13th 2010 on Millennium Village Blog. "Consider an example. The MVP introduces agricultural support programs in Year 1 to provide 100kg of subsidized fertilizer and 10kg of modern seeds to 10,000 local farmers, in a country where most farmers use less than 10kg of fertilizer per household and very few have access to modern seeds. In the first season, the MVP farmers increase their yield by 2 tons per hectare. The national government then decides that it wants to provide similar support to every smallholder farm household in the country, and is successful in mobilizing ODA support to do so, but the country can only afford to subsidize 50kg to each farmer along with 5kg of seeds. The result is that the national average yield increases by 1 ton per hectare." What is the real effect in MVP site due to MVP intervention? 1 ton or 2 tons yield increase Would much appreciate your thoughts on the question as I am trying to think about it myself.

Thanks for the comment. Check out the first part of our reply to the MVP, where we tried to address this issue: http://blogs.worldbank.org/africacan/evaluating-the-millennium-villages-reply-to-the-mvp-upcoming-seminar-with-comments-from-jeff-sachs Fundamentally, for any impact evaluation, the relevant question is what would have happened in the counterfactual, i.e. if the project/intervention had never taken place. We argue that what took place in country as a whole (or, as an alternative, the area around the MVP site) during the MVP period is a reasonable, though imperfect, estimate of the counterfactual of the MVP. The fact that some other interventions that are similar to some components of the MVP, e.g. provision of subsidized fertilizer, may have taken place in other places is entirely irrelevant for estimating the *effects of the MVP package*, which is the question at hand. That's because those interventions would have taken place even if the MVP had never existed.

Submitted by Raj Raina on
Many thanks for referring me to the link. Your argument seems reasonable. But at the same time, Earth Institutes' response (below) to the above questions seems just as reasonable. Why is not ok to simply report on progress and let people do their own subtractions and additions to the results? (Because as you mention the counterfactual is not perfect and as we know, development/progress in not inevitable. Than, it is quite possible that a village might have seen reverse development without MV interventions and now using the same logic MV would be underreporting its results.) "In this example, a simple difference-in-difference model could be falsely interpreted as showing that that the MVP interventions increased the yield by only 1 ton, and that to claim a 2 ton increase overstates the results of the interventions. This is obviously an erroneous conclusion. The MVP intervention (100kg of fertilizer and 10kg of seed) is working exactly as advertised: raising yields by 2 tons. Outside of the villages, half the intervention package is producing half the results. There is no “over-reporting” of results, merely an increased treatment dose in the MV, which is exactly the correct point." MV blog

I disagree with the suggestion that the MV example presented here undermines the difs-in-difs approach. Again, the crucial question is *What would have happened at the project site if the MVP had never existed?* This is ultimately unknowable, because we never observe the project site in this state of the world--that's why it's called the counterfactual. But it is impossible to meaningfully define "impact" without identifying what an estimate of the counterfactual is, so we have to come up with a some estimate. It's true that without the MVP, things at the MVP site might have gotten worse. By the same taken, without the MVP, things might have improved *more* than they did with the MVP. What we're looking for is not the full distribution of possible counterfactual states, but rather what is our best estimate of what the counterfactual is. A very good--though not perfect--estimate of the counterfactual is what happened in rural areas in the same region as the MVP site. This is the counterfactual behind the difs-in-difs estimates. The hypothetical fertilizer and seed example confuses the issue by suggesting that the question is one of analyzing simple interventions which are also taking place elsewhere. But the evaluation question is not "What is the impact of free fertilizer?" Rather, it is "What is the impact of the MVP package of interventions?" The fact that programs similar to some components of the MVP may have been implemented elsewhere does not complicate the question of evaluating the impact of the full package.

Raj, I really appreciate your thoughtful comment because it cuts to the heart of the issue. I would like to illustrate the question you raise with an analogy. Suppose a group of people generally takes aspirin for headaches. We start a program to get them to take Tylenol instead. Part of the group is encouraged to take Tylenol for headaches and does so; the rest of the group keeps doing what they were doing before: taking aspirin for headaches. What is the 'effect' of what we're doing? There are two ways to define it. If you want to know the impact *of Tylenol*, you would want to compare pain in people who took Tylenol to pain in people who took nothing. You might find that Tylenol cures headaches, and works "exactly as advertised", in the words of the MVP. If you want to know the impact *of our intervention*, you would want to compare pain in the people we encouraged to take Tylenol to pain in people who kept doing whatever they were doing before we came along---taking aspirin. You might find that there's no difference in pain between the two groups, because both Tylenol and aspirin reduce pain effectively. Thus the effect *of our intervention* on pain is nil, even though Tylenol works "exactly as advertised" to reduce pain. Thus we have two different questions and two different answers. One is not "obviously" more correct than the other. But in the context of the MVP, one is much more useful than the other. There is no doubt that fertilizer raises crop yields. That is well established. We do not need an evaluation of that. There *is* important doubt about whether the MVP causes crop yields to go up more than they would have gone up due to whatever would have happened without the MVP. Going back to the analogy: We know Tylenol works, so the first form of the question is not useful to pose. We don't know whether our particular Tylenol promotion campaign makes people have less pain; that *does* need evaluation. We are extremely clear at every point in our paper that we seek to measure the impact *of the project, as a project*. So I am baffled by the MVP's immutable confusion on this straightforward point.

Submitted by Anonymous on
Re Michael Clemens's post: Well-posed causal questions (of the sort we are discussing, a la "Rubin Causal Model") are always:
**What is the effect of X-intervention (treatment) vs. Y-intervention (control) on Z (the dependent variable) for a given set of units (individuals, firms, etc.)?**
The point that Michael is trying to make would be easier and clearer communicated if he were to frame his causal questions being explicit about the Y-intervention. The title of Clemens's post notwithstanding, the word "effect" has one meaning (a well-defined meaning in this context, see the "Rubin" papers going back to the 1970s) which can be applied to different causal questions with clarity and transparancy when those questions are explicitly articulated.

The Millennium Villages Project has stated that one of the "impacts" or effects of its project was to raise primary school enrollment by a specified quantity at some of the sites. It has never modified or retracted that claim. There is no ambiguity about what that means. It means that without the project, primary school enrollment would not have risen by that specified quantity at that site. The project's statement certainly does not mean, to any reasonable person, that "enrollment increased at the site due to larger forces independent of our project". But the fact that enrollments have been increasing all across some of the large areas where the small MVP sites are located means that it is likely that much or all of the increase seen at the intervention sites would have occurred if the project had never existed. That makes it baseless to assert that the "impact" of the project is the full change observed at the site, which the project has done for years and continues to do. This is a simple point. No sophisticated models are required to grasp it. Just an interest in genuine impact.

Submitted by olugbenga adesanya on
Most of the comments on MVP and her impacts appear to beg the issue. The question should be if the MVP initiatives meets the real intention of the promoters and if it could bring development in real terms to poor communities within sub-Sahara Africa. Let us appraise the MVP model, retain the benefits and correct the negativity. We should stop this academic debate and move towards food security and agro-based industrial economies to avoid Africa ending up being a left behind continent.

Submitted by Raj Raina on
Firstly, I agree that food security is critical for not only countries in Africa but also for all countries in the world. Food security today is a global security issue. Stopping the debate over what works and what does not, will not solve the problem of food security but may exacerbate it due to incorrect interventions. Evaluation and the "academic debate" can be used in a variety of ways. For instance: •to help make resource allocation decisions •to help rethink the causes of a problem •to support decision making on competing or best alternatives •to build consensus on the causes of a problem and how to respond. I also agree MVP is doing fantastic work (I spent 3 months in a MV3) but at the same time I am curious to know whether the model should be scaled up? If so how? And for that you need evaluation/debate.

As we discuss in our paper, the Southwest Poverty Reduction Project in China spent $580 million on a huge, similar five-year village-level package intervention, with no lasting impact on the incomes of low-income people. That is hardly an "academic" problem. Diverting vast amounts of aid money to a project, without carefully establishing that the project can meet its own stated goal of sparking "self-sustaining economic growth", has a direct effect on the lives of real people. Calling it "academic" does not change that fact.

Submitted by olugbenga adesanya on
Dear Micheal, As an International Project Manager, I know that the success or failure of projects are dependent project definition and pre-contract analysis et al. Apologies if you take exception to the phrase 'academic'. Regards Gbenga

Add new comment