Last week on Let’s Talk Development, I asked what the term “science of delivery” (SOD) means. I suggested that SOD is about moving from thinking about “what to deliver” to “how to deliver”. We know, for example, the interventions that cut child mortality (bednets, vaccinations, breastfeeding, etc.) but these interventions reach too few children, and the trick is to get them delivered to more. Much of the Bank’s analytic work, policy dialogue and lending work has focused precisely on how to reform policies and programs to ensure the interventions that are needed to improve development outcomes actually reach people. Much of this work merits the term “science” – it makes use of an explicit “theory of change” in the form of a results framework that reflects the latest social science, and builds on rigorous empirical evidence that compares actual outcomes with an explicit and plausible counterfactual.
It turns out this isn’t the only way the term “science of delivery” is used. Last week the World Bank had a visit from Michael Barber who headed Tony Blair’s Delivery Unit in the UK, and is author of “Deliverology 101”. Barber and Blair joined World Bank Group president Jim Yong Kim on stage for a conversation on the science of delivery and “deliverology” (they mean the same thing in Barber’s mind); Barber subsequently gave a seminar that I attended, and met with two of the sector boards that I sit on (Public Sector, and Education).
The intriguing thing is that Barber’s “science of delivery” isn’t the “science of delivery” I blogged about last week. Nevertheless, it’s an interesting one, and it’s worth reflecting on how the two SODs fit together with one another and with the Bank’s work.
SOD à la Barber
As Blair put it, it’s one thing to get elected and make promises on the basis of broad policies; it’s another to deliver on these promises. The deliverology approach starts by ensuring that there are clear priorities, and that associated with each are a limited number of specific, measurable and ambitious targets. (Barber didn’t mention it but in the UK hospital executives and head teachers lost their jobs for falling short of certain targets.) The approach clarifies – and if necessary tweaks – the implementation plan; the plan doesn’t have to be 100% right at the start, just good enough to get started. Other key elements to deliverology are: having data that arrive on a frequent basis (monthly if not weekly); and having in place a team that can analyze these high-frequency data to see whether the targets are going to be hit. There need to be regular and routine reviews of progress involving top officials (in the UK, Blair regularly reviewed progress). These “stocktakes” need to be honest (the top official needs to learn the truth) and the conversations about what the problems are and how to solve them need to be honest. Barber advises never ignoring a problem once it’s identified.
The process needs to be relentless and the “deliverology team” has to be isolated from the everyday crises of government. The deliverologists claim that this approach yielded dramatic improvements in crime, education, and health in the UK, and in education in Pakistan.
Challenges to and from the deliverologists
At the end of the day, “deliverology” is really about improving policy implementation; it basically takes policy as given. Interestingly the team in the British government that deals with this agenda now is called the “Implementation Unit” not the “Delivery Unit”. Delivery clearly includes the implementation of policies that promise better outcomes, but it must also include the design of policy itself. Thus “implementation science” or “implementationology” might be a more honest term than “deliverolgy”. That said, deliverology throws down a challenge: how much of delivering better development outcomes is about better policy, and how much of it is about better implementation? Barber claims that 90% is about better implementation.
Deliverology also throws down a challenge specifically to the Bank. In its lending and non-lending work, the Bank works upstream of implementation – it helps design and then “supervises” lending operations (projects) but the client government implements the projects; and the Bank helps design and then monitors the policies its loans support, but the government implements the policies. On the face of it, then, this branch of the science of delivery lies well outside the purview of the Bank.
But does it? The impact of the Bank’s aid depends on how well Bank projects and Bank-supported policy reforms are implemented. In a way what the deliverologists are saying is that the Bank shouldn’t take as given the scope for transforming projects and policies into development outcomes, and that the Bank might do more (perhaps a lot more) to help governments do implementation better. Given the Bank’s traditional arms-length involvement in implementation, an obvious question (which I won’t try to answer) is: how might this happen?
Another point that emerged – as was commented on by many during the two days of deliverology – is the importance of high-level commitment. (In the seminar we learnt of the failure of the Kuwati Delivery Unit because the government never “owned” it.) Also important are local capacity and financial resources – having on the ground 24/7 a group of highly driven target-oriented number-crunchers and problem-solvers. (I noticed in the report on Pakistan that the team were mostly with McKinsey, and most were expatriates.) These are two pretty strong prerequisites that are rarely satisfied in Bank client countries.
A third point that struck me is that we don’t seem to know much about the impacts of the deliverology approach. There are data on outcomes before and after the application of deliverology, but it’s not clear what we can infer from them. In Pakistan’s Punjab province we know, as Shanta Deverajan pointed out in his comments, that enrollment as increasing at a fast pace even before the application of deliverology. Education and health outcomes improved in England when deliverology was being applied and stagnated in Wales over the same period where deliverology wasn’t being applied, but there were several bold policy changes that occurred in England during the period that didn’t occur in Wales. In neither case, then, is it clear how much of the improvement can be attributed to the application of deliverology rather than to policy change. And as others have pointed out, and several Bank staff wondered during the visit, there are serious risks of gaming with such a target-driven approach. (To his credit Barber met this head on in the Q&A, and said he always asked skeptics to tell him what to monitor in addition to the target indicators so he could detect gaming.)
Last, I kept looking for but could never quite find the “science” in deliverology. How scientific were the principles were that were used to problem-solve when targets were at risk of not being hit? In Pakistan, for example, the deliverology team spotted that in the village they visited only a few girls were attending school. The team persuaded these girls to escort the non-attending girls to school, and recommended to the government that this approach be used province wide. The team deserves some credit for quick on-the-ground thinking, and the approach has the merit of not requiring a change of policy. But was it sustainable? Effective? Cost-effective? More so than other alternative approaches to getting girls into school, many of which have been subjected to rigorous evaluations and are based on a theory of change informed by one or more social sciences? For example, we know from a World Bank impact evaluation that the Punjab Female School Stipend Program raised enrolments among girls by nine percent – effects that subsequent Bank research showed persisted five years into the program.
I’m not claiming that every action needs to entail a policy change and needs to be based on a fully fledged social science-based theory of change with accompanying rigorous evidence based on a credible counterfactual. But without some of this some of the time, there’s surely a serious risk of resources not being allocated efficiently in the pursuit of better development outcomes. And that, after all, is what SOD is ultimately about.
Join the Conversation