Evaluating the evaluating of the Millennium Villages Project


This page in:

not all millennium projects are this neatly contained within clearly defined bordersWhen is the rigorous impact evaluation of development projects a luxury, and when is it a necessity?

This is a question asked in a new paper examing the Millennium Villages Project (MVP), a high profile initiative that, according to its web site, offers a "bold, innovative model for helping rural African communities lift themselves out of extreme poverty".

In the words of one of the authors of When Does Rigorous Impact Evaluation Make a Difference? The Case of the Millennium Villages, "We show how easy it can be to get the wrong idea about the project’s impacts when careful, scientific impact evaluation methods are not used. And we detail how the impact evaluation could be done better, at low cost."  The paper underscores the importance of comparing trends identified within a project activity with those in comparator sites if one is to determine the actual impact of a specific project.  This sentiment should come as no surprise to those familiar with an area of exploding interest in the international donor and development community -- that of the usefulness of randomized evaluations.

While project proponents, critics and neutral observers alike would all agree that MVP is an ambitious work in progress (with many supporters likely ascribing a dual meaning to this characterization), lessons from this experience are already being formulated by policymakers and advisors in many places. The impact of the MVP project is not only important for the participating communities in Kenya (and the other project sites), but in many other places as well, as this is seen by many as one possible model for a sustained, long-term engagement to support economic development of potential broader relevance. It also provides opportunities for very concrete and contemporary explorations of how to evaluate the impact of an intervention of this sort at scale.  (International donor agencies like the World Bank are of course particularly interested in "scaling up").

For those with a particular interest in "ICT for development" (ICT4D) or "ICT in education",  the Millennium Villages Project (and the related Millennium Promise initiative) has been of particular interest, given its high profile in exploring how mobile phones, village connectivity and other ICT-related tools and solutions can relevant to the interlinked challenges of health, infrastructure, education and economic development in poor village communities. 

As interesting as the report itself is, those time-pressed people who profess to not to read full papers (I know there are more than a few people who characterize themselves this way -- read into that what you wish) may prefer to scan a set of related blog posts that have come out since this paper was released.  To me, these provide a very quick, real-life primer on evaluation issues -- not only on methodologies and approaches, but also, perhaps (if you read between the lines) on some of the related attendant politics (and posturing).

The paper: When Does Rigorous Impact Evaluation Make a Difference? The Case of the Millennium Villages (CGD Working Paper 225)

Note: There will be a related debate at the World Bank in DC on Wednesday, 27 October, at 2pm DC time, which will be livestreamed over the Internet.

Some time in 2011 we expect to see first results from a rigorous, IDB-funded randomized evaluation of the OLPC program in Peru, one of the very few attempts to date to apply RCT methodologies to an educational technology initiative in a developing country.  It will be interesting to see how many of the themes from recent exchanges around the Clemens/Demombynes paper will re-surface in what I expect will be lively commentary around the evaluation of another quite high profile development project. (Those interested in this upcoming evaluation report may wish to monitor the IDB's ICT in education blog for updates.)

Please note: The photo at the top of this blog post of London's Millennium Dome, as seen from the Isle of Dogs ("not all millennium projects are this neatly contained within clearly defined borders"), was taken by Jan van der Crabben; it comes via Wikimedia Commons and is used according to the terms of its Creative Commons Attribution-Share Alike 2.0 Generic license.



Michael Trucano

Global Lead for Innovation in Education, Sr. Education & Technology Policy Specialist

Michael Trucano
October 26, 2010

A quick follow-up:
The seminar on "Evaluating the Millennium Villages" announced for October 27th has been postponed. It will be re-scheduled and available online. Those interested in this event are advised to monitor Gabriel Demombynes's blog, http://blogs.worldbank.org/africacan/blogs/gabriel-demombynes and/or Michael Clemens's Twitter feed, http://twitter.com/m_clem.
I will do my best to post an update as a comment here as well.

Michael Clemens
October 26, 2010

Gabriel Demombynes and I have posted a reply to the statement of the Millennium Village Project on our paper. We explain why much of that statement rests on a basic misunderstanding of fundamental concepts in project impact evaluation. You can read it here: http://bit.ly/dvVWaG

Michael Trucano
March 28, 2011

Here's a video from a public discussion on evaluating the Millennium Villages Project from Oxford University’s Centre for the Study of African Economies (CSAE) --> last week http://goo.gl/esAns

Michael Trucano
October 19, 2011

In case it might be of any interest to readers of the blog post above:

My World Bank colleague David McKenzie published a blog post today with the title "Jeff Sachs, the Millennium Villages Project, and Misconceptions about Impact Evaluation" that makes for interesting reading:

This is in response to a blog post by Jeffrey Sachs & Prabhjot Singh,

which was in itself a response to a blog post by Lawrence Haddad of IDS,

For good measure, Madeleine Bunting on the Guardian's Poverty Matters blog has also weighted in, http://goo.gl/inwcx .

(Sorry for using so many link shorteners, the original URLs are quite long and wreak havoc on the default table layouts in the WB blog platform.)