After years of bad news from developing countries about high rates of health worker absenteeism, and low rates of delivery of key health interventions, along came what seemed like a magic bullet: financial incentives. Rather than paying providers whether or not they show up to work, and whether or not they deliver key interventions, doesn’t it make sense to pay them—at least in part—according to what they do? And if, after doing their cost-benefit calculations, women decide not to have their baby delivered in a health facility, not to get antenatal care, and not take their child to be immunized, then doesn’t it make sense to try to change the benefit-cost ratio by paying them to do so?
Financial incentives in health have taken off
Much of the developing world has said Yes to the first question, and many developing countries have said Yes to the second as well. Donors have agreed. Since 2007, they’ve given $436 million to the World Bank to spend on projects that help governments move away from budgets and salaries (monies that get paid irrespective of performance) to a mix of budgets, salaries and bonus payments linked to results. Most of the $436 million are supporting pilot performance-based financing (PBF) projects in low-income countries. The funds are leveraging $2.4 billion to support project implementation from the International Development Association, the arm of the World Bank that provides concessional lending to low-income countries.
Financial incentives on the demand side have also been introduced—for example, payments to women who deliver their babies in a public facility. Sometimes, programs have financial incentives on both the supply and demand sides: incentivizing providers may not help much if families find visiting public facilities too costly; and incentivizing families may not help much if providers don’t show up to work.
Skimpy evidence to date
So far, despite the widespread hope that financial incentives are a magic bullet for developing countries, and despite the large financial commitments, the evidence is still pretty skimpy.
Two reviews of demand- and supply-side incentives, one by the Cochrane Collaboration, the other by the German Development Institute, concluded that little could be said definitively from the 21 programs studied, partly because the evaluation methods were weak. A more recent review of demand-side measures was more upbeat, but still pointed out lots of areas where we know too little.
The World Bank’s PBF projects will help fill the knowledge gap. All but eight of the 42 projects (six of which are standalone studies of middle-income programs) are scheduled to produce an impact evaluation, many using a prospective randomized control trial (RCT) design. To date, however, only three have done so. They found mixed results: in one country (DRC) no impacts were seen; in the others (Argentina and Rwanda), positive impacts were found, but not on all indicators. The other 31 PBF projects will produce useful evidence, but we won’t get the results for months, and in many cases years.
Partly because it’s going to take a while before we see the results, we probably shouldn’t rely exclusively on these and other prospective studies. It would be good to know more now about what works rather than wait several years. After all, now is when countries are looking to make some pretty big investments.
There’s another reason not to pin all our hopes on these prospective studies: it’s likely there will be quite a few questions they won’t give us answers to. We won’t have a good sense of whether the results will hold as the program is taken from the experimental phase and scaled up nationally. And it’s likely there will be some important design questions that these experiments won’t be able to address—because the experiment is too small, or because the resources aren’t available, or because the questions are considered ‘second-order’, and the people responsible for the projects are keen to focus on what they see as the ‘first-order’ question of whether financial incentives ‘work’.
Can’t we learn more now by doing more and better retrospective studies?
Three recent papers take a different tack to the prospective RCT. And while they don’t conclude that financial incentives in health do not work, they do suggest that financial incentives aren’t quite the magic bullet many hoped they would be.
These studies are retrospective studies of at-scale (or almost at-scale) programs. RCTs ‘identify’ program effects through randomized assignment to different ‘treatment’ groups. Retrospective studies of at-scale programs can’t use that ‘identification strategy’. What they can do, though, is exploit the fact that these programs weren’t implemented overnight everywhere, but instead rolled out first in some places, then others, and so on. By comparing changes in outcomes over time between households already exposed to the program and those not yet exposed, adjusting for differences in ‘covariates’ between the already ‘treated’ and the as-yet ‘untreated’, researchers are able to estimate the effects of the programs on key outcomes.
One of these retrospective studies looks at PBF in Cambodia—the first developing country ever to do PBF (the first scheme started in 1999). Cambodia is also the country that has tried the most ‘flavors’ of PBF. Sometimes, depending on the time periods and locality, PBF has been accompanied by demand-side incentives, and sometimes the MOH has contracted with NGOs either to provider care or manage MOH facilities. All PBF programs in Cambodia have specified performance targets for antenatal care (ANC), delivery in a public facility, childhood immunization and the use of birth-spacing methods.
The second study looks at Burundi whose PBF program aimed to improve maternal and child health outcomes, and was rolled out across the country between 2006 and 2010 in three waves. PBF payments account for around 40% percent of a facility’s revenue, and are linked to various output indicators including ANC, vaccinations and family planning.
The third study looks at the developing world’s largest demand-side incentive program: the Janani Suraksha Yojana (JSY) scheme in India, which provides cash to women who give birth in a public health facility. The amount varies between urban and rural areas, and between states, but the sum paid is typically at least as much as women usually pay to have their baby delivered in a government facility (around $25). (Women are supposed in principle to be able to get the cash if they deliver in an approved private facility, but in practice this part of the scheme hasn’t yet started working.) The program also provides a cash payment to accredited health workers who attend a delivery in a public facility.
Call it magic?
Looking down the columns of the table below where the results of three studies are summarized, it’s clear that India’s demand-side program comes the closest to delivering what it was supposed to—getting women, especially poor women, to deliver in a facility. It also encouraged breastfeeding, which wasn’t incentivized by the program. Despite this, however, JSY had no effect on neonatal mortality. This is significant given today’s report on child mortality from UNICEF, WHO and the World Bank, which concluded that the neonatal period remains the biggest challenge in reducing under-five mortality. The report found that deaths in the first 28 days now account for 45% of deaths among under-fives, and that neonatal mortality has fallen more slowly than postnatal under-five mortality (47% compared with 58%).
The two PBF programs did less well. There’s only one unambiguously good result—Burundi’s PBF scheme raised immunization rates, and achieved bigger increases (for most types of vaccination) among the poor. The rest of the table is mixed news, or plain bad news. The mixed news is that in Burundi PBF had some impact on ANC content but not on ANC visits, and that in both Burundi and Cambodia PBF raised government-facility deliveries but only among the nonpoor. The plain bad news is that PBF in Cambodia basically had no impacts on ANC, facility deliveries overall, immunization, or neonatal mortality—this despite the fact that several of the PBF schemes received a higher level of spending per capita.
Why not so magic?
PBF design matters. The Cambodia study suggests that the structure of incentives and the degree of autonomy providers have likely makes a difference—results were worse under a halfway-house arrangement where an external contractor had some but not complete autonomy; in places where this didn’t happen, even if the contracting was internal to the MOH, impacts were more likely to be found.
Demand-side and supply-side incentives work on different margins—demand-side incentives encourage people to go to a facility, while supply-side incentives encourage health providers to deliver more and better care to people who have made it to the facility. This helps explain why in Cambodia PBF worked better when accompanied by a maternity care voucher scheme, why in Burundi PBF affected the content of ANC but not the number of visits, and why in India the demand-side JSY scheme had such a large effect on facility deliveries, including among the poor. Demand- and supply-side incentives are complements, and are best combined; the evidence from these studies suggests that if only one is feasible, it might be better to go with the demand-side approach.
Crowd-out is underappreciated. In Cambodia and India, the scheme crowded out some private-sector facility deliveries. It’s not at all obvious this is a good thing. The government is using scarce resources to subsidize something people are willing to pay for, and may not be delivering better quality services. (Jeff Hammer has a nice blog post on this theme.) Standardized patients (the gold standard for assessing quality of care) haven’t been used to assess the quality of care in deliveries—unsurprisingly. But standardized-patient evidence from India for other casetypes suggests that private providers there are significantly more likely to ask the right questions and do the right exams, even if they are not more likely to prescribe the right treatment. We should at least be documenting the crowd-out and doing quality comparisons.
Last, given we’re ultimately concerned about outcomes, we need to secure impacts at all points along the results chain. In all three studies there was little or no effect on ANC. Yet we know that steps taken prior to childbirth can improve the survival prospects of neonates. There were effects on facility deliveries, but it may well be, as Jishnu Das has argued in a recent blog post, that the facilities we’re encouraging people to use through these schemes aren’t equipped to deliver the relevant intra- and postpartum interventions. In fact, they may be less well equipped than the private facilities the Cambodia and India schemes steered women away from. And further down the results chain, it’s likely that the various schemes didn’t do enough to increase the delivery of key community-based neonatal interventions. Using financial incentives to relieve one bottleneck isn’t much good if bottlenecks remain further up or down the results chain.
Thanks Adam for this blog. You are right to point out that PBF design matters. The Cambodia PBF schemes were very early experiments, learning lessons on design and implementation, which lessons have led to ongoing experimentation and ultimately to the successful Rwanda scale up (and successful IE results) and Burundi scale-up (the most recent scale-up and most advanced PBF approach to date). I do not know about the Indian scheme, and although results seem impressive, I would not advice to go for a single condition, a ‘vertical target’ in any PBF scheme. These are some of the many design and implementation lessons learned over a 15 year period https://openknowledge.worldbank.org/handle/10986 /17194.
On ANC….we sometimes overemphasize ANC whereas what really matters is the final result: that there is less morbidity and mortality. Morbidity and mortality related to childbirth are tackled chiefly through Emergency Obstetric Care and not through ANC. And this is why a holistic approach with adequately defined service packages is so important.
And finally….let us avoid looking at PBF or RBF as a ‘magic bullet’, but instead let us talk about health financing and the health system reforms necessary to get better results. There is no magic bullet to get better results, unless if you equate the ‘magic bullet’ to wholesale well-financed well-designed and well-implemented health reforms. We have to reform the system to get it to produce better results in a more sustainable manner. And reforms are never magic bullets, are never a one-off event, as they are incremental, and continuous. And we all agree here don’t we?
Gyuri, thanks for the comment. Just to clarify: the papers, which it sounds like you haven't read, look not only at the pilot phases in Cambodia and Burundi but at the entire experience -- pilot and post-pilot. Which is why their results are so important -- and sobering. Cheers, Adam
Thank you for this very helpful summary, especially about securing impacts along the results chain. You mention that both Cambodia and Burundi had incentives for family planning. Could you share the results on family planning as well?
Hi Priya, if I recall correctly, FP wasn't analyzed in the papers, but you can click on the links to see for yourself. Cheers, Adam
Adam - Good post. Besides pointing out that the advocacy for these approaches is running ahead of the evidence, it is also worth highlighting examples of countries that have done very well without them, thank you very much. Sri Lanka is perhaps the best example - no RBF, no CCT, yet IMR and U5MR are anywhere from 25 to 50 years ahead of most comparators in both East and South Asia. Its experiences may not translate elsewhere, but at least let's have the conversation before suggesting that incentives are somehow essential.
Thx, Osmith. Fair point. Of course, it might be the case that (well designed) financial incentives in Sri Lanka could make the country's performance even more spectacular. Right now we don't know that's not the case. Cheers, Adam
Thanks Adam for a thoughtful and thought provoking blog. A few responses:
1) PBF may not have worked but Contracting with NGOs did. A prospective quasi-experiment in 12 districts in Cambodia between 1999 and 2003 showed very large changes in population level coverage of antenatal care, vitamin A, immunization, and facility deliveries. The double differences were very large, about half a standard deviation on average. The slander against these results is that they involved larger public investments in the experimental districts compared to control districts. However, these expenditures were more than offset by reduced out of pocket expenditures.
2) Something went right in Cambodia: DHS surveys in 2000 and 2010 show a 56% decline in U5MR (from 124/1000 to 54) and the maternal mortality ratio declined 53% (from 437-206/100,000). This may have been due to the competition between different approaches and the foment that was engendered by an openness to new ways of doing things.
3) Not all RCTs are created equal: To conclude that the RCT in DRC offsets the positive results from Argentina and Rwanda is to neglect the dreadful implementation of PBF in DRC. As a stakeholder in that debacle, it is a little naive to conclude that the results shed doubt on PBF. It's like saying that a plumber who does open heart surgery demonstrates the lack of impact of the surgery.
Thx, Benjamin, for the comment. Pls do take a look (when you get a VPN connection!) at the Cambodia paper. It references but goes beyond (in terms of scope and methods) previous work. The paper drills down on each of the various stages of PBF in Cambodia, and among other analyses replicates the analysis of the 1999-2003 pilots. It does find effects on deliveries, but finds no effects on vaccinations or (for the most part) on ANC. So a different story from yours. The paper also finds no effects of PBF on neonatal mortality, so I think it would be a stretch to claim the decline in U5MR you mention could be due to PBF, no? Fair point on the three prospective studies. I was just flagging they had been done and felt I should mention the findings. The post was really about the three retrospective studies and what we can learn from them. Thx again, Adam
Thanks, Adam. This is interesting. When the RBF ‘pilots’ started few years ago, it was based on the belief (not necessarily hard evidence) that paying health providers for what they do makes sense. I want to believe that the impact evaluation that accompanied almost all of these RBF programs financed by HRITF is in recognition of this fact and to fill this gap in knowledge. Hopefully the BODY of evidence form these impact evaluation will guide the future of RBF. For now the evidence from the few countries seems to support that RBF is indeed not the magic bullet that we were hoping for.
I also want to raise somewhat related issue. What does the future looks like for RBF? Given that financial contributions from donor financed RBF programs are meager compared to the domestic budget for health, should we envision RBF as a catalyst for reform in health financing in general (the payment system, in particular)? If so, what is next for RBF programs?
Thanks, Tekabe. Glad to see you're keen that evidence should drive policy reform. I think you're right in that these programs are best seen as catalysts for payment reform. I guess the question we need to consider now in the light of this new evidence is: What form should that payment reform take? Cheers, Adam
I agree that it’s not only about the cash…..but if we want to focus on the cash for a minute why not focus on the majority of the money with the most potential to be truly pro-poor? The majority of the general revenue funding flowing through less than optimal health purchasing/provider payment mechanisms obstructed by public finance management rigidities? And why promote that RBF or P4P (or PBF or PBB or the next flavor of the month…..) are separate or different from other output-based provider payment systems?
Compare it to your own job. It’s like saying you have no financial incentive to show up at work every day or “perform” to get the salary that represents 80-90% of your compensation but you’re going to jump through hoops to “perform” to get 10-20% of your compensation? And that this bonus comes from a completely different company with no connection to your employer and its general HR mechanisms? Wouldn’t you wonder why you bother with that annual performance review?
Back to my normally optimistic “big dream” thinking……unless you can manufacture money, the bottom-line to me is that all provider payment has always had and always will have financial incentives, it’s easier and sexier to separate and control your own RBF, P4P, etc. experiment, but it seems that the real road to UHC is digging in to do the heavy lifting of health purchasing/provider payment and PFM reform, harmonization and alignment. Or at least to think in terms of mixed models or relating all provider payment systems where RBF or P4P can leverage all money or enable implementation sequencing for broader health purchasing and PFM reforms.
Thanks, Sheila. I guess most payment systems are mixed or blended systems, e.g. mixing salaries with incentive payments. What you’re saying is that if the P4P component is too small a part of such a system, and P4P works as intended, it’s not going to have much ‘bite’. It’s less obvious to me that payments from different sources will have different effects on a provider just because they’re from different sources, though clearly from a management perspective, the ideal would be to have one management structure for all payments. Cheers, Adam
Dear Adam, I really enjoyed reading your blog. I am currently pursuing a Master´s Degree in Health Economics and Management at the Erasmus University Rotterdam, and our professor recommended us to read your blog. I am trying to extrapolate the Asian cases to my context in Latin America. I live in Paraguay, and many specialists in the country consider the PBF as the solution to the absenteeism problem. However, I believe that even though a reform is in its way, the majority of the work force will not accept the PBF solution. On the contrary, as the country is mainly covered by the national health system and nearly every professional works in the public sector (but also in the private sector), they will reject the idea and lobby against any possible reform that incorporates PBF. The only possible solution might be a progressive (parallel to the old model) implementation with the new professionals hired by the system. About the demand side, I believe that community empowerment (and especially promotion/prevention actions through the community) might work better than financial incentives, considering the idiosyncrasies of the people. Lastly, I would say that an experiment (RCT) would be very useful to make conclusions on the topic in my particular community. I hope someday I will have the opportunity to do so and contribute with recommendations. Cheers from Paraguay!