Syndicate content

evaluation

Millennium Villages Project continues to systematically overstate its effects

Gabriel Demombynes's picture

by Michael Clemens and Gabriel Demombynes

The Millennium Villages Project (MVP) is an experimental anti-poverty interventionin villages across Africa. In October, we released evidence that the Project’s official publications were overstating its real effects, and we offered suggestions on improving its impact evaluation. On Tuesday the MVP, whose leadership and staff are aware of our work, continued to greatly overstate its impact.

Evaluating the Millennium Villages: Reply to the MVP + Upcoming Seminar with Comments from Jeff Sachs

Gabriel Demombynes's picture

The following post was co-authored by Michael and Gabriel.

The Millennium Village Project (MVP) is an important, experimental package of interventions that the United Nations and Columbia University are testing in 14 villages across Africa. The MVP offers a tremendous opportunity to learn whether such interventions can catalyze self-sustaining growth and escape from extreme poverty. But the evaluation approaches currently being used cannot generate convincing evidence of the Project’s impacts. Without such evidence, it will be impossible to generate the billions of dollars needed to scale up the Project approach across Africa, as its proponents hope to do.

We have written a new research paper (summarized here and here) that proposes small and inexpensive modifications to the MVP evaluation approach that would make it possible to evaluate the Project’s impacts.

That paper has generated much discussion, including reports in the Financial Times and in a major newspaper in Kenya. The Project itself has issued a lengthy official response by Pronyk, McArthur, Singh, and Sachs. We welcome this public debate as a way to improve learning about what works in development. We answer below the main questions posed in the Project’s response, much of which rests on a basic misunderstanding.

Evaluating the Millennium Villages

Gabriel Demombynes's picture

Here’s the quick summary of a new working paper I have co-authored with Michael Clemens of the Center for Global Development:

When is the rigorous impact evaluation of development projects a luxury, and when a necessity? We study one high-profile case where it is a necessity: the Millennium Villages Project (MVP), an experimental intervention in rural Africa. We compare development trends inside versus outside the villages in three countries, and show that estimates of the project’s effects depend heavily on the evaluation method.

The impact evaluation currently planned by the MVP is unlikely to yield adequate estimates of its effects on Africans in general, for five reasons we explain. But it is not too late to carefully measure the project’s effects, by making small and inexpensive changes to the next wave of the project.

Michael’s own blog post gives more details about the paper. The paper uses publicly-available data from the MVP mid-term evaluation report and Demographic and Health Surveys  (DHS). Field visits played no role in the study.

But after the study I found myself wanting to learn more about a couple of the places behind the statistics. So after we completed the analysis, during September 26-28, I took a trip with several World Bank colleagues to the western edge of Kenya. We visited two village clusters in Nyanza Province: first the MVP site in Bar-Sauri, and then the town of Uranga, 50 km to the west, which is not an MVP site.

Here’s a picture of me pressing the flesh with the kids at Nyamninia Primary School in Bar-Sauri: