Syndicate content

Millennium Villages Project

Poor Evaluation Methods Can Mislead: New Developments in the Millennium Villages Evaluation

Gabriel Demombynes's picture

by Michael Clemens and Gabriel Demombynes

Contrary to persistent perceptions that sub-Saharan Africa is mired in intractable misery, many of the region’s countries have experienced sustained economic growth, deepening democracy, improving governance, and decreasing poverty in recent years.

To take just one aspect of the African Renaissance, in five of six countries for which recent data is available—Malawi, Tanzania, Rwanda, Nigeria, and Ghana—rates of child malnutrition as measured by stunting have declined in the last decade. Because so much is changing in Africa, it is crucial to take this “background” change into account when evaluating the impact of local policy interventions.

This is evident when considering the Millennium Villages Project (MVP) evaluation, which we critiqued in a peer-reviewed journal article. Recently, we examined the three peer-reviewed papers that dealt with the MVP’s impacts and showed that they do not back up the project’s claims of large impacts, in part because they don’t take “background” change into account.

There’s a new development: The MVP has just released its first study that does try to distinguish changes observed at its village sites from broader changes happening across Africa.

MVP evaluation session at Oxford

Gabriel Demombynes's picture

I am at Oxford for the annual conference of the Center for Study of African Economies, which runs through Tuesday.

Here's the program with links to many of the conference papers.

Plenary sessions, including my presentation on the Millennium Village Project evaluation, will be broadcast live on the web, and the recording will later be posted on the CSAE website.

I'll present a short version of the paper, which was co-authored with Michael Clemens of the Center for Global Development. This will be followed by a presentation from John McArthur, CEO of the Millenium Promise organization. Our session will be the last of the conference, on Tuesday 6-7 p.m. UK time (2-4 pm East Coast U.S. time.) 

For background, here are the first, second, third, and fourth earlier posts on the paper and check out our podcast, the MVP response, and commentary from Julian Jamison, Chris Blattman, Eric Green, and Bill Easterly and Laura Freschi.

Here's also video of an extended talk on the MV paper which Michael and I gave in DC in December:

Evaluating the Millennium Villages: Reply to the MVP + Upcoming Seminar with Comments from Jeff Sachs

Gabriel Demombynes's picture

The following post was co-authored by Michael and Gabriel.

The Millennium Village Project (MVP) is an important, experimental package of interventions that the United Nations and Columbia University are testing in 14 villages across Africa. The MVP offers a tremendous opportunity to learn whether such interventions can catalyze self-sustaining growth and escape from extreme poverty. But the evaluation approaches currently being used cannot generate convincing evidence of the Project’s impacts. Without such evidence, it will be impossible to generate the billions of dollars needed to scale up the Project approach across Africa, as its proponents hope to do.

We have written a new research paper (summarized here and here) that proposes small and inexpensive modifications to the MVP evaluation approach that would make it possible to evaluate the Project’s impacts.

That paper has generated much discussion, including reports in the Financial Times and in a major newspaper in Kenya. The Project itself has issued a lengthy official response by Pronyk, McArthur, Singh, and Sachs. We welcome this public debate as a way to improve learning about what works in development. We answer below the main questions posed in the Project’s response, much of which rests on a basic misunderstanding.