by Michael Clemens and Gabriel Demombynes
Contrary to persistent perceptions that sub-Saharan Africa is mired in intractable misery, many of the region’s countries have experienced sustained economic growth, deepening democracy, improving governance, and decreasing poverty in recent years.
To take just one aspect of the African Renaissance, in five of six countries for which recent data is available—Malawi, Tanzania, Rwanda, Nigeria, and Ghana—rates of child malnutrition as measured by stunting have declined in the last decade. Because so much is changing in Africa, it is crucial to take this “background” change into account when evaluating the impact of local policy interventions.
This is evident when considering the Millennium Villages Project (MVP) evaluation, which we critiqued in a peer-reviewed journal article. Recently, we examined the three peer-reviewed papers that dealt with the MVP’s impacts and showed that they do not back up the project’s claims of large impacts, in part because they don’t take “background” change into account.
There’s a new development: The MVP has just released its first study that does try to distinguish changes observed at its village sites from broader changes happening across Africa.
That new paper, just published in the American Journal of Clinical Nutrition (AJCN), examines changes in child stunting over three years at MVP sites in nine countries. The paper observes that 1) on average, stunting in those countries showed a flat trend at the national level before the project began, and 2) on average, stunting fell at the MVP sites during the program period (3 years starting in 2005/6). From these trends during different time periods, the project concludes that the decline in child malnutrition at the sites is a “result” and “achievement” of the project.
But comparing trends at the MVP sites to nationwide trends in two different time periods is a critical mistake. Although child nutrition did not generally improve in sub-Saharan Africa over the two decades before the project started, it did generally improve during the period the project was in place. This suggests that some improvement would have taken place at the MVP sites even if the project had never been implemented. Comparing changes on the sites in one period to changes off the sites in a different period, as the AJCN paper does, likely tends to overstate the impacts of the project at the sites.
Furthermore, trends within the regions where MVP sites are located are a better comparison point than national trends. We performed this comparison for the MVP sites in Ghana, Nigeria, and Kenya. In all three, the Millennium Village site is located in a part of the country where stunting is falling faster than nationwide. Here is the picture for Ghana showing fraction of children stunted over time:
In rural areas of the Ashanti region where the MVP site is located, stunting has been falling just as much as at the project site. The Millennium Village contains less than 1% of Ashanti Region’s population, so even allowing for a generous “spillover” effect of MVP programs to neighboring areas, it is implausible that the village is driving the trend across Ashanti. Comparing the project site to the national trend is likely to overstate the impact of the project.
The AJCN paper has other weaknesses. Most importantly, the paper says almost nothing about how the individual village sites were chosen for project participation. In an earlier response to our paper, the MVP rejected “any insinuation that the Millennium Villages were somehow systematically advantaged at the outset of the project.” However, the project’s own research protocol says, “Issues of feasibility, political buy-in, community ownership and ethics also featured prominently in village selection,” and according to one independent account (PDF),
“Half of the research villages have been situated in areas with an established history of interaction with foreign development groups. The idea is that the likelihood of rapid success is greater in places such as these because inexperience with and cultural barriers against development work have already been overcome.”
If this is the case, then successes at those sites have limited value for predicting the project’s potential in other areas.
Although the project cites the AJCN article and other articles to support its claims of impact, these papers do not rigorously support those claims. We raise these points not because we believe the MVP has had no effect at all but because, as we argue in the paper, we believe that rigorous and persuasive impact evaluation is essential to the mission we share with the Millennium Villages Project of seeking to understand how to spur growth and reduce poverty in the developing world.
Join the Conversation