Syndicate content

A new approach to measuring the impact of global health aid

Cristian Baeza's picture

AV17-33 World Bank

We in the global health community have been successful during the past decade in advocating for additional funding for health. We saw huge increases in development assistance for health, and our work has attracted support from political leaders, celebrities, and taxpayers alike. But now in 2011 we face a number of challenges, with donor governments facing significant fiscal pressures to reduce their aid budgets, and increasing scrutiny on how we measure the real impact of these investments.

Last week I was pleased to brainstorm with several of our key health partners to discuss what we can do.

During the past few years, the World Bank and leading global health agencies have relied on a fairly simple but compelling metric–lives saved–to demonstrate the impact of our work. Of course, talking about lives saved is compelling, and it has helped make global health work better understood by non-health experts. But there are a few challenges with this approach thus far.

First, the way we currently measure lives saved tends to downplay the importance of a well-functioning health system in saving a person's life. Measuring lives saved as a result of specific commodities such as a bednet, vaccine, or antiretroviral treatment only tells us one part of the impact story. For these commodities to be effective, we need the full value chain of a good health system, including the motivated and trained health worker, the well-equipped clinic, the cold-chain storage, the affordable financing, clean water and infrastructure, and the right policies, logistics and more–all of these things must be in place to achieve the desired health impact.

So the bottom line is: We need a better way to measure our results. We need to take more of a “value chain” approach, looking at all the different elements in a health system that contribute to one healthy life. We also must be much more rigorous in our estimates, measuring changes in health outcomes attributable to each of the various parts of the health value chain. To start, it makes sense to focus on what it takes to achieve the health Millennium Development Goals. We are starting work on this new approach, and I’ll keep you apprised of our progress periodically. Please stay tuned–and I hope you will join us.

Comments

Metrics are a double-edged sword. As mentioned in the post, focusing on one such as lives saved tends to take away the focus from other items, such as value chain. In one of my blog posts, I noted that they seem to function like squeezing a balloon - it gets smaller in the area where we are focusing but is getting larger where we are not. They are a useful tool, but they are not a substitute for human judgment and common sense. We need to be careful not to attribute a status to them that they are unable to carry.

Thank you very much for your feedback. I agree. Metrics such as lives saved (or any other) are an instrument to communicate, and results or performance should not substitute for an integral view of the value of the program(s). In development in health, we continue to have the challenge that metrics tend to be complex to communicate to non-health experts. Thus, the need for more compelling metrics. Your point is very important. Many thanks.

Submitted by Pete Kolsky on
Cristian, I'm not sure you actually responded to David's conclusion, which was "[Metrics] are a useful tool, but they are not a substitute for human judgment and common sense. We need to be careful not to attribute a status to them that they are unable to carry." This is not a problem that a "more compelling metric" can address. When I first started working professionally 35 years ago, the evaluation and monitoring agenda was just beginning, and the need for it was clear.... it seemed as if decision-making in development had completely ignored "evidence" as a basis for decision-making. I believe David's point, (and I agree with it,) is that the drive for "evidence"(of a certain quantitative and statistical nature) has swung so far that one can end up overloading the limits of what metrics can achieve, at the expense of development. This is like the story of the guy looking for a lost ring under a streetlamp at night, even though he knows it rolled off to the side, because "that's where the light is"; the temptation to focus on "those results which are clearly and easily measured" as opposed to "those results which are important" is great. This is never an easy balance...I don't advocate going after important goals without worrying about trying to measure progress towards them. But I am worried, as I believe David is, that judgment and common sense are under-valued, and that many are hoping that "with just the right metric" or "evaluation study" all will be clear. Put another way, if the human race had refused to invest in activities until they had a metric to assure its success, we would probably never have moved beyond the hunter-gatherer mode.

Submitted by Pete Kolsky on
I fully agree with the blog emphasis on value chains, and not just outcomes. In water supply we learned a while ago, (but are in danger of forgetting), that rather than JUST focusing on health impacts, it was smarter to work our way up the results chain or value chain... Q1: Were the water supplies built? If not, done; If they were built, go to Q2: Are they operating properly? If not, done. If so, go to Q3: Are people using it? (which people? How? Why or why not?) If they are using it, THEN it makes sense to worry about health (and other) impacts, which are among the hardest to measure. The advantage of such an approach is that you gather an awful lot of operationally useful info answering questions 1 through 3, even if you don't come up with a clear answer on 4. And if you do come up with a clear answer on 4, (either more positive or negative), then you have some data with which to hypothesize explanations.

Add new comment