Syndicate content

Evaluating the Millennium Villages

Gabriel Demombynes's picture

Here’s the quick summary of a new working paper I have co-authored with Michael Clemens of the Center for Global Development:

When is the rigorous impact evaluation of development projects a luxury, and when a necessity? We study one high-profile case where it is a necessity: the Millennium Villages Project (MVP), an experimental intervention in rural Africa. We compare development trends inside versus outside the villages in three countries, and show that estimates of the project’s effects depend heavily on the evaluation method.

The impact evaluation currently planned by the MVP is unlikely to yield adequate estimates of its effects on Africans in general, for five reasons we explain. But it is not too late to carefully measure the project’s effects, by making small and inexpensive changes to the next wave of the project.

Michael’s own blog post gives more details about the paper. The paper uses publicly-available data from the MVP mid-term evaluation report and Demographic and Health Surveys  (DHS). Field visits played no role in the study.

But after the study I found myself wanting to learn more about a couple of the places behind the statistics. So after we completed the analysis, during September 26-28, I took a trip with several World Bank colleagues to the western edge of Kenya. We visited two village clusters in Nyanza Province: first the MVP site in Bar-Sauri, and then the town of Uranga, 50 km to the west, which is not an MVP site.

Here’s a picture of me pressing the flesh with the kids at Nyamninia Primary School in Bar-Sauri:

 

Here’s me arriving at the Gongo Health Centre in Bar-Sauri with my colleague Fred Owegi and being greeted by medical officer Danstan Ochien and MVP acting team leader Jessica Masira:

 

The facility was built and is paid for by the MVP. The clinic is well-stocked with free drugs provided by the MVP, and is staffed by the medical officer, three nurses, and a medical technician paid for by the project. It’s evident that there have been improvements in Bar-Sauri in recent years.

Next we headed to Uranga, which did not receive the MVP intervention. In Uranga we visited schools, a shopping area, and the Helmi Jones Health Centre.  Here is a picture of my colleague Anne Khatimba at the entrance:

 The clinic's facilities look similar to those we saw at the clinic in Bar-Sauri. The facility is funded largely through its modest fees but also receives support from the Catholic Church and a government grant program. Sister Anita Odera, who heads the facility and is clearly a deeply dedicated and warm-hearted woman, told us that since 2006, she has seen an increase in the rates of child immunization and HIV testing, and that more births now take place at the clinic rather than in homes.

These and other anecdotes suggest that it’s worth asking how trends inside the Millennium Villages reported by the MVP compare to trends outside the intervention sites. But anecdotes can mislead, so in the paper we compare data from the Millennium Village sites reported by the MVP to data from large-scale DHS surveys carried out across the surrounding region and country.

Here are graphs from the paper showing changes at the Millennium Village site in Bar-Sauri, compared to trends in rural Nyanza, rural Kenya, and Kenya overall:

 

  

 I've included the three measures mentioned by Sister Odera and also mobile phone ownership because the MVP mid-term evaluation report highlights “Proportion of households that own a mobile phone increased fourfold” as one of the project’s “biggest impacts” in Bar-Sauri.

These graphs make it clear why comparing the before-and-after situation in Bar-Sauri, as the MVP mid-term evaluation does, may give an exaggerated sense of the project’s effects.  The blue lines show that all four of these indicators were on upward trends in rural Nyanza, the province where Bar-Sauri is located. This strongly suggests that the assumption that no change would have taken place in Bar-Sauri in the absence of the MVP intervention—the assumption behind impact estimates based on before-and-after comparisons—is wrong.  In the paper we calculate how much the impact estimates might change if broader trends outside the Millennium Villages were taken into account, both in Bar-Sauri and at two other Millennium Village sites in Ghana and Nigeria.

The MVP midterm evaluation report notes that at some point in the future it will release impact evaluations to do somewhat better than simple before-and-after comparisons, by studying selected comparison villages. That evaluation protocol has several limitations, however—many of which are acknowledged in the protocol itself—and I’ll leave it to Michael’s post to sketch those.

Comments

Submitted by Ad Waijers on
The medical and educational projects of Dr. Bea Andersen on the slopes of Mount Elgon in Western Kenya, 40 km from the city of Kitale are a great example of the huge impact on the population in terms of health and education and is an excellent field of research for the millenium villages, set up by a commercial company Mount Elgon Orchards. http://www.mtelgon.com/ http://www.sokitale.nl/sok/en http://www.elimumountelgon.nl/english-version_41.html

Submitted by Ruud Ross on
When we started in 1996 with the SOK http://www.sokitale.nl/ we could never imagine to have so much impact on the community around Endebess on the slopes of Mount Elgon. The idea from Dr Bea Andersen's father to start with a small foundation to help people in need ended years later in a complete new village plan: http://www.sokitale.nl/Kitale/kitale.html As you can imagine we are excited and also proud about it! We keep on "building" to provide more support in health, education and try to reach the millenium goals. http://www.mtelgon.com/ http://www.sokitale.nl/sok/en http://www.elimumountelgon.nl/english-version_41.html

Submitted by Kyu Lee on
Please see the response from the Millennium Villages project to the Clemens and Demombynes paper. Available here: http://2mp.tw/50

Submitted by Anonymous on
Kudos for putting spotlight on this rather "clumsy" effort at developing a village model of development. I'm from nearby village in Sauri and agree that besides the evaluation methodology there are some serious development gaps with MV model: 1. As the paper correctly points out, there are some villages within the province that might have done even better than MVP. 2. My problem with MVP and in particular Sauri project is that it is not scalable, not to a level where all nearby villages can perhaps be incoporated into the project, or, at the least, be in a position to assimilate or undertake similar model. I mean the project has been in Sauri for a quite a number of years, but it seems like it is only concentrated on this small village that is sorrounded by abject poverty. 3. Sustainability - can this project really be sustainable without external resources? The difficult to sustainability goes to point #2 - scalable. 4. Invovlement of local leaders/national leaders/other international organziation- this project seems to be so selective in regards to involving leaders from this province as well as national leaders/ministries within Kenya government. For example, how is the ministry of agriculture in Kenya involved in the project? How about legislators from the region - beyond Sauri. Again thank you for writing a wonderful paper. I know that folks at MVP are going to debunk it but I have been there and as the nearby community knows, this project cannot be scaled to the level where we can actually say that it is making significant impact of lives of people beyond the tiny village of Sauri.

Above, commenter Kyu Lee of the Earth Institute links to the official statement of the Millennium Village Project on our research paper. In our reply to that statement, my co-author and I explain why much of the statement rests on a basic misunderstanding of fundamental concepts in project impact evaluation. You can read our reply here: http://bit.ly/dvVWaG

Add new comment