Syndicate content

millennium development villages

Poor Evaluation Methods Can Mislead: New Developments in the Millennium Villages Evaluation

Gabriel Demombynes's picture

by Michael Clemens and Gabriel Demombynes

Contrary to persistent perceptions that sub-Saharan Africa is mired in intractable misery, many of the region’s countries have experienced sustained economic growth, deepening democracy, improving governance, and decreasing poverty in recent years.

To take just one aspect of the African Renaissance, in five of six countries for which recent data is available—Malawi, Tanzania, Rwanda, Nigeria, and Ghana—rates of child malnutrition as measured by stunting have declined in the last decade. Because so much is changing in Africa, it is crucial to take this “background” change into account when evaluating the impact of local policy interventions.

This is evident when considering the Millennium Villages Project (MVP) evaluation, which we critiqued in a peer-reviewed journal article. Recently, we examined the three peer-reviewed papers that dealt with the MVP’s impacts and showed that they do not back up the project’s claims of large impacts, in part because they don’t take “background” change into account.

There’s a new development: The MVP has just released its first study that does try to distinguish changes observed at its village sites from broader changes happening across Africa.

MVP evaluation session at Oxford

Gabriel Demombynes's picture

I am at Oxford for the annual conference of the Center for Study of African Economies, which runs through Tuesday.

Here's the program with links to many of the conference papers.

Plenary sessions, including my presentation on the Millennium Village Project evaluation, will be broadcast live on the web, and the recording will later be posted on the CSAE website.

I'll present a short version of the paper, which was co-authored with Michael Clemens of the Center for Global Development. This will be followed by a presentation from John McArthur, CEO of the Millenium Promise organization. Our session will be the last of the conference, on Tuesday 6-7 p.m. UK time (2-4 pm East Coast U.S. time.) 

For background, here are the first, second, third, and fourth earlier posts on the paper and check out our podcast, the MVP response, and commentary from Julian Jamison, Chris Blattman, Eric Green, and Bill Easterly and Laura Freschi.

Here's also video of an extended talk on the MV paper which Michael and I gave in DC in December:

Millennium Villages Project continues to systematically overstate its effects

Gabriel Demombynes's picture

by Michael Clemens and Gabriel Demombynes

The Millennium Villages Project (MVP) is an experimental anti-poverty interventionin villages across Africa. In October, we released evidence that the Project’s official publications were overstating its real effects, and we offered suggestions on improving its impact evaluation. On Tuesday the MVP, whose leadership and staff are aware of our work, continued to greatly overstate its impact.

Evaluating Millennium Villages Revisited

Shanta Devarajan's picture

Although the members of the Millennium Village Project were unavailable (but have offered to hold a follow-up seminar in January), we held a seminar on the Clemens-Demombynes  paper to discuss different approaches to evaluating rural development programs.

Evaluating the Millennium Villages

Gabriel Demombynes's picture

Here’s the quick summary of a new working paper I have co-authored with Michael Clemens of the Center for Global Development:

When is the rigorous impact evaluation of development projects a luxury, and when a necessity? We study one high-profile case where it is a necessity: the Millennium Villages Project (MVP), an experimental intervention in rural Africa. We compare development trends inside versus outside the villages in three countries, and show that estimates of the project’s effects depend heavily on the evaluation method.

The impact evaluation currently planned by the MVP is unlikely to yield adequate estimates of its effects on Africans in general, for five reasons we explain. But it is not too late to carefully measure the project’s effects, by making small and inexpensive changes to the next wave of the project.

Michael’s own blog post gives more details about the paper. The paper uses publicly-available data from the MVP mid-term evaluation report and Demographic and Health Surveys  (DHS). Field visits played no role in the study.

But after the study I found myself wanting to learn more about a couple of the places behind the statistics. So after we completed the analysis, during September 26-28, I took a trip with several World Bank colleagues to the western edge of Kenya. We visited two village clusters in Nyanza Province: first the MVP site in Bar-Sauri, and then the town of Uranga, 50 km to the west, which is not an MVP site.

Here’s a picture of me pressing the flesh with the kids at Nyamninia Primary School in Bar-Sauri: