Evaluating Millennium Villages Revisited
This page in:
Although the members of the Millennium Village Project were unavailable (but have offered to hold a follow-up seminar in January), we held a seminar on the Clemens-Demombynes paper to discuss different approaches to evaluating rural development programs. My reaction to the presentation and discussion is that there really shouldn’t be a controversy here. It’s true that the original paper by the MVP, “Harvests of Development” does not have a properly specified counterfactual (what would have happened in the absence of the intervention, including the service delivery innovations of the project) and therefore does not constitute an evaluation. The Clemens-Demombynes (CD) paper attempts to specify a counterfactual by comparing the performance of MVP villages with that of neighboring and similar villages. While not perfect, this procedure meets many of the criteria for a reasonable evaluation. In fact, as we discussed at the seminar, it avoids some of the possible “contamination effects” of a seemingly more rigorous randomized evaluation.
But the point is that the CD paper shows that, relative to their counterfactual, the MVP performs well along a number of dimensions, such as improved sanitation, measles vaccination, and reducing child mortality. Not surprisingly, it does not perform well in cell phone usage in Kenya, a country where mobile phones have spread throughout the country. In short, Clemens and Demombynes have undertaken the first evaluation of the MVP. They have shown that the MVP has delivered sizeable improvements on some important development indicators in many of the villages, albeit with effects that are smaller than those described in the Harvests of Development paper. Of course, neither study answers the question of whether these gains are sustainable, or whether they could have been obtained at lower cost. These should be the subject of the next evaluation.
Did the counterfactual take into consideration the fact that non-MVP villages could also have been getting some external assistance too, either from government or some NGO or other donors? And also that such assistance was not going to the project villages because of the presence of MVP? It is not an impossibility than a non-MVP village receives more aid than an MVP village.
The ideal conterfactual is that which was completely secluded from any external assistance, not even government.
I have not read the paper by CD yet. I just need some clarifications from C and D.
In addition to the cost issues raised by Shanta, perhaps the most important question is whether this "experiment" in a small number of carefully supported and nurtured villages can be scaled-up in any meaningful and cost-effective way through a country-led process? As we all know, there is s huge difference between efficacy (demonstrated impact in a research set-up)and effectivness at scale. Coverage, capacities, governance, quality of implementation, ownership and costs...all have a major role to play in determining this gap between efficacy and effectiveness.
Hi Shanta,
Thank you for sharing your insight. I always enjoy them.
I was wondering why is the evaluation method of Millennium Villages receiving attention now from WB staff?
The Millennium Villages have been active for over 5-years. According to Earth Institute website to date, the Millennium Villages project has reached nearly 400,000 people in 79 villages. Clustered into 12 groups across 10 African countries (Ethiopia, Ghana, Kenya, Malawi, Mali, Nigeria, Rwanda, Senegal, Tanzania, and Uganda).
Given the MV is a project from the Earth Institute at Columbia University and importantly the United Nations Development Program, shouldn't World Bank staff have had prior knowledge of the villages and been able to provide feedback earlier?
On slightly different topic but relevant question, how does the World Bank and United Nations share information about projects? (Do the PRSPs cover the villages?)
kind regards,
Raj
The paper is not the first evaluation of the MVP: the Overseas Development INstitute in London conducted an evaluation, available here:
http://www.odi.org.uk/work/projects/details.asp?id=765&title=millennium…
Thanks for your comment, Raj. The evaluation method of the MVP is receiving attention (not just by World Bank staff, but by others in the development community I might add) mainly because the results of the project are only now being registered. The Harvest of Development paper was the first report to disseminate results from the projects. This is the case with most development projects--they have to operate for a while before results can be measured.
Information about World Bank projects are available on our website (www.worldbank.org). In addition, our Independent Evaluation Group publishes its evaluation of World Bank projects on its website (http://www.worldbank.org/ieg/).
Firstly I just wonder why the experment of Millennium villages is at all necessary as no body will miss the fact that locally implmented perpetual develpment schems together with good governance will bring about improvement in the living situation of the impoverished. I say this because there is similar story i remember that i saw many many years back. A genuine finish mission project in a rural africa produced one of a village mimicking the garden of aden with pharmacy offering all sorts of europian medicine, manicured enviroment and empowered villagers. Some what in a decade or half, the donor fund run out of bank and the project managers had to pull out leaving the place to the local government. Less than a year was enough to go back to zero or squre number one.
The lesson seems whatever solution for poverty eradication should come locally and be supported internationally if support exists. Africa is suffering from poverty of governance which needs addressing before addressing poverty in itself which the donor community does not care much about. Unless that is stemmed out, all effort of donor money, it comes arround and it goes or turns arround back to donors or dictators.
woyabid