Syndicate content

Poor Evaluation Methods Can Mislead: New Developments in the Millennium Villages Evaluation

Gabriel Demombynes's picture

by Michael Clemens and Gabriel Demombynes

Contrary to persistent perceptions that sub-Saharan Africa is mired in intractable misery, many of the region’s countries have experienced sustained economic growth, deepening democracy, improving governance, and decreasing poverty in recent years.

To take just one aspect of the African Renaissance, in five of six countries for which recent data is available—Malawi, Tanzania, Rwanda, Nigeria, and Ghana—rates of child malnutrition as measured by stunting have declined in the last decade. Because so much is changing in Africa, it is crucial to take this “background” change into account when evaluating the impact of local policy interventions.

This is evident when considering the Millennium Villages Project (MVP) evaluation, which we critiqued in a peer-reviewed journal article. Recently, we examined the three peer-reviewed papers that dealt with the MVP’s impacts and showed that they do not back up the project’s claims of large impacts, in part because they don’t take “background” change into account.

There’s a new development: The MVP has just released its first study that does try to distinguish changes observed at its village sites from broader changes happening across Africa.

That new paper, just published in the American Journal of Clinical Nutrition (AJCN), examines changes in child stunting over three years at MVP sites in nine countries. The paper observes that 1) on average, stunting in those countries showed a flat trend at the national level before the project began, and 2) on average, stunting fell at the MVP sites during the program period (3 years starting in 2005/6). From these trends during different time periods, the project concludes that the decline in child malnutrition at the sites is a “result” and “achievement” of the project.

But comparing trends at the MVP sites to nationwide trends in two different time periods is a critical mistake. Although child nutrition did not generally improve in sub-Saharan Africa over the two decades before the project started, it did generally improve during the period the project was in place. This suggests that some improvement would have taken place at the MVP sites even if the project had never been implemented. Comparing changes on the sites in one period to changes off the sites in a different period, as the AJCN paper does, likely tends to overstate the impacts of the project at the sites.

Furthermore, trends within the regions where MVP sites are located are a better comparison point than national trends. We performed this comparison for the MVP sites in Ghana, Nigeria, and Kenya. In all three, the Millennium Village site is located in a part of the country where stunting is falling faster than nationwide. Here is the picture for Ghana showing fraction of children stunted over time:

In rural areas of the Ashanti region where the MVP site is located, stunting has been falling just as much as at the project site. The Millennium Village contains less than 1% of Ashanti Region’s population, so even allowing for a generous “spillover” effect of MVP programs to neighboring areas, it is implausible that the village is driving the trend across Ashanti. Comparing the project site to the national trend is likely to overstate the impact of the project.

The AJCN paper has other weaknesses. Most importantly, the paper says almost nothing about how the individual village sites were chosen for project participation. In an earlier response to our paper, the MVP rejected “any insinuation that the Millennium Villages were somehow systematically advantaged at the outset of the project.” However, the project’s own research protocol says, “Issues of feasibility, political buy-in, community ownership and ethics also featured prominently in village selection,” and according to one independent account (PDF),

“Half of the research villages have been situated in areas with an established history of interaction with foreign development groups. The idea is that the likelihood of rapid success is greater in places such as these because inexperience with and cultural barriers against development work have already been overcome.”

If this is the case, then successes at those sites have limited value for predicting the project’s potential in other areas.

Although the project cites the AJCN article and other articles to support its claims of impact, these papers do not rigorously support those claims. We raise these points not because we believe the MVP has had no effect at all but because, as we argue in the paper, we believe that rigorous and persuasive impact evaluation is essential to the mission we share with the Millennium Villages Project of seeking to understand how to spur growth and reduce poverty in the developing world.
 

Comments

Submitted by Jared Osoro on
This is an extremely interesting anayisis that debunks nearly a whole thesis espoused by Jefrrey Sachs in his book "The End of Poverty: Economic Possibility for our Time". Oh Jeffrey! Oh Bono!

Submitted by M. Duban on
I think this places great emphasis on importance of having control sites, already after choosing the implementation sites in a methodological manner.

Submitted by S. Ling on
“Half of the research villages have been situated in areas with an established history of interaction with foreign development groups. The idea is that the likelihood of rapid success is greater in places such as these because inexperience with and cultural barriers against development work have already been overcome.” I belive that this statement sums up the same thing that Fellow Colleagues, Demombynes and Clemens said about the MVP Report. That there would need to be more “Proof” to access certain bottom line statements such as the quote above. Right?

Submitted by Vivan U. Ukah on
Dear Gabriel, I find this read very interesting and I am impressed by the reports. I strongly agree with you that proper evaluation methods have to developed and utilised to get the actual facts right. This has been a challenge in developing countries due to poor reporting systems. However, it is still very achievable and would go a long way in planning and implementing effective programmes so as to eradicate poverty.

Add new comment