Evaluating the evaluating of the Millennium Villages Project

When is the rigorous impact evaluation of development projects a luxury, and when is it a necessity?

This is a question asked in a new paper examing the Millennium Villages Project (MVP), a high profile initiative that, according to its web site, offers a "bold, innovative model for helping rural African communities lift themselves out of extreme poverty".

In the words of one of the authors of When Does Rigorous Impact Evaluation Make a Difference? The Case of the Millennium Villages, "We show how easy it can be to get the wrong idea about the project’s impacts when careful, scientific impact evaluation methods are not used. And we detail how the impact evaluation could be done better, at low cost."  The paper underscores the importance of comparing trends identified within a project activity with those in comparator sites if one is to determine the actual impact of a specific project.  This sentiment should come as no surprise to those familiar with an area of exploding interest in the international donor and development community -- that of the usefulness of randomized evaluations.

While project proponents, critics and neutral observers alike would all agree that MVP is an ambitious work in progress (with many supporters likely ascribing a dual meaning to this characterization), lessons from this experience are already being formulated by policymakers and advisors in many places. The impact of the MVP project is not only important for the participating communities in Kenya (and the other project sites), but in many other places as well, as this is seen by many as one possible model for a sustained, long-term engagement to support economic development of potential broader relevance. It also provides opportunities for very concrete and contemporary explorations of how to evaluate the impact of an intervention of this sort at scale.  (International donor agencies like the World Bank are of course particularly interested in "scaling up").

For those with a particular interest in "ICT for development" (ICT4D) or "ICT in education",  the Millennium Villages Project (and the related Millennium Promise initiative) has been of particular interest, given its high profile in exploring how mobile phones, village connectivity and other ICT-related tools and solutions can relevant to the interlinked challenges of health, infrastructure, education and economic development in poor village communities. 

As interesting as the report itself is, those time-pressed people who profess to not to read full papers (I know there are more than a few people who characterize themselves this way -- read into that what you wish) may prefer to scan a set of related blog posts that have come out since this paper was released.  To me, these provide a very quick, real-life primer on evaluation issues -- not only on methodologies and approaches, but also, perhaps (if you read between the lines) on some of the related attendant politics (and posturing).

The paper: When Does Rigorous Impact Evaluation Make a Difference? The Case of the Millennium Villages (CGD Working Paper 225)

• blog post by Michael Clemens (co-author, CGD)
• blog post by Gabriel Demombynes (co-author, World Bank)
• blog post response from the MVP team (including Jeffrey Sachs):
• related articles in Kenya's Daily Nation newspaper, the Financial Times (by ex-IFC staffer Tim Harford) and over at Aid Watch.

Note: There will be a related debate at the World Bank in DC on Wednesday, 27 October, at 2pm DC time, which will be livestreamed over the Internet.

Some time in 2011 we expect to see first results from a rigorous, IDB-funded randomized evaluation of the OLPC program in Peru, one of the very few attempts to date to apply RCT methodologies to an educational technology initiative in a developing country.  It will be interesting to see how many of the themes from recent exchanges around the Clemens/Demombynes paper will re-surface in what I expect will be lively commentary around the evaluation of another quite high profile development project. (Those interested in this upcoming evaluation report may wish to monitor the IDB's ICT in education blog for updates.)

Please note: The photo at the top of this blog post of London's Millennium Dome, as seen from the Isle of Dogs ("not all millennium projects are this neatly contained within clearly defined borders"), was taken by Jan van der Crabben; it comes via Wikimedia Commons and is used according to the terms of its Creative Commons Attribution-Share Alike 2.0 Generic license.