Evaluating the Millennium Villages

Here’s the quick summary of a new working paper I have co-authored with Michael Clemens of the Center for Global Development:

When is the rigorous impact evaluation of development projects a luxury, and when a necessity? We study one high-profile case where it is a necessity: the Millennium Villages Project (MVP), an experimental intervention in rural Africa. We compare development trends inside versus outside the villages in three countries, and show that estimates of the project’s effects depend heavily on the evaluation method.

The impact evaluation currently planned by the MVP is unlikely to yield adequate estimates of its effects on Africans in general, for five reasons we explain. But it is not too late to carefully measure the project’s effects, by making small and inexpensive changes to the next wave of the project.

Michael’s own blog post gives more details about the paper. The paper uses publicly-available data from the MVP mid-term evaluation report and Demographic and Health Surveys  (DHS). Field visits played no role in the study.

But after the study I found myself wanting to learn more about a couple of the places behind the statistics. So after we completed the analysis, during September 26-28, I took a trip with several World Bank colleagues to the western edge of Kenya. We visited two village clusters in Nyanza Province: first the MVP site in Bar-Sauri, and then the town of Uranga, 50 km to the west, which is not an MVP site.

Here’s a picture of me pressing the flesh with the kids at Nyamninia Primary School in Bar-Sauri:

 

Here’s me arriving at the Gongo Health Centre in Bar-Sauri with my colleague Fred Owegi and being greeted by medical officer Danstan Ochien and MVP acting team leader Jessica Masira:

 

The facility was built and is paid for by the MVP. The clinic is well-stocked with free drugs provided by the MVP, and is staffed by the medical officer, three nurses, and a medical technician paid for by the project. It’s evident that there have been improvements in Bar-Sauri in recent years.

Next we headed to Uranga, which did not receive the MVP intervention. In Uranga we visited schools, a shopping area, and the Helmi Jones Health Centre.  Here is a picture of my colleague Anne Khatimba at the entrance:

 The clinic's facilities look similar to those we saw at the clinic in Bar-Sauri. The facility is funded largely through its modest fees but also receives support from the Catholic Church and a government grant program. Sister Anita Odera, who heads the facility and is clearly a deeply dedicated and warm-hearted woman, told us that since 2006, she has seen an increase in the rates of child immunization and HIV testing, and that more births now take place at the clinic rather than in homes.

These and other anecdotes suggest that it’s worth asking how trends inside the Millennium Villages reported by the MVP compare to trends outside the intervention sites. But anecdotes can mislead, so in the paper we compare data from the Millennium Village sites reported by the MVP to data from large-scale DHS surveys carried out across the surrounding region and country.

Here are graphs from the paper showing changes at the Millennium Village site in Bar-Sauri, compared to trends in rural Nyanza, rural Kenya, and Kenya overall:

 

  

 I've included the three measures mentioned by Sister Odera and also mobile phone ownership because the MVP mid-term evaluation report highlights “Proportion of households that own a mobile phone increased fourfold” as one of the project’s “biggest impacts” in Bar-Sauri.

These graphs make it clear why comparing the before-and-after situation in Bar-Sauri, as the MVP mid-term evaluation does, may give an exaggerated sense of the project’s effects.  The blue lines show that all four of these indicators were on upward trends in rural Nyanza, the province where Bar-Sauri is located. This strongly suggests that the assumption that no change would have taken place in Bar-Sauri in the absence of the MVP intervention—the assumption behind impact estimates based on before-and-after comparisons—is wrong.  In the paper we calculate how much the impact estimates might change if broader trends outside the Millennium Villages were taken into account, both in Bar-Sauri and at two other Millennium Village sites in Ghana and Nigeria.

The MVP midterm evaluation report notes that at some point in the future it will release impact evaluations to do somewhat better than simple before-and-after comparisons, by studying selected comparison villages. That evaluation protocol has several limitations, however—many of which are acknowledged in the protocol itself—and I’ll leave it to Michael’s post to sketch those.