I recently finished reading Chris Blattman’s new book “Why we fight: the roots of war and the paths to peace”. It is very well-written, interspersing interesting anecdotes and personal experiences with a great overview of literature from political science, economics, history, and psychology. The first half of the book uses a simple game theory bargaining framework along with some insights from psychology to argue that most of the time groups should be able to settle differences without fighting, and then to outline five different categories of reasons as to why fighting may occur. The second half of the book then discusses what types of policy actions do and do not seem promising for reducing the likelihood of fighting. The coverage seems reasonably comprehensive, but this is not my area of expertise, so I’m not sure whether there are competing explanations in the literature not discussed – my sense is that he is better at discussing why wars are more or less likely to break out, and less about what can be done about them when they are underway, and what causes them to stop – which also may reflect the relative amount of evidence about these. I found it a convincing read overall, and he is able to communicate a lot of ideas succinctly in a way that should be easy to read for non-academics.
Tinker and trial before writing the manual and grant application
I wanted to highlight for our readers a few paragraphs from his conclusions on the path forward, which I think will resonate a lot more generally with our readers at the World Bank and with others planning projects or attempting to test new interventions. He urges a piecemeal approach, of attempting to get improvement at the margins, and one of his key pieces of advice is the need for organizations to learn from mistakes, and to expect trial and error to also involve error. He writes (p289-290):
“Almost every big project I’ve seen started the same way: a mayor, a government ministry, or an aid agency comes up with an idea; they write up a program manual outlining who will get what; they get funding from the central government or an outside donor; their approach gets laid out ahead of time in the grant proposal; they immediately implement at scale, to thousands or tens of thousands of people. A hundred things go wrong. They tweak the rollout to fix the worst problems. But the core design never changes, no matter how flawed. Three years into their five-year grant, the implementors have little idea if the program is working, and they are beset by nagging worries that it is middling at best. Everyone involved knows about the mediocrity, but they’re fearful to admit it. Their organizations insulate them and the project from accountability.
Every time I see this happening, I suggest the same thing: “Hold on. Why don’t we figure out what we want to achieve and come up with five or six different ways we could do this. Roll them all out rough. Pilot for three or four months. Tinker. Watch. Collect some data. Interview people. Then we write the program manual with the one or two best ideas, finalize the design with the donor, and roll it out to a larger group.” It doesn’t have to take more time. In a five-year plan, what better way can you think of to spend years one and two? And it doesn’t mean spending more money. In fact, it costs less when you don’t toss funds at mediocre solutions for years. I have made this argument dozens of times - that the path to peace and prosperity needs to be discovered through piecemeal tinkering. Few take me up on it, from Colombia to Uganda to Liberia to Chicago. It’s like watching a car crash in slow motion every time.”
The challenges of doing this in practice
I’ve had this exact same conversation multiple times as well. In a lot of the interventions I’m thinking about, it can take a while to see outcomes emerge, and so I’m doubtful that in 3-4 months we can find out whether something is working in many cases. This is one of the constraints facing a lot of development interventions in thinking about whether to apply adaptive trials. But 3-6 months can certainly be enough time to learn that something is not working as intended, and to improve designs. In particular, a lot of interventions that sound great in theory suffer from one of the following problems:
(a) Low demand/no take-up: we may be able to learn quite quickly that no one really wants to use/take-up our fantastic new program, despite what they might have said when asked hypothetically in focus groups. Tinkering may help boost take-up, we may need to change the targeting or outreach for the program, or it may be that there are other more binding constraints that need to be addressed first.
(b) Implementation issues: an idea that sounds great on paper may be too complex and difficult to roll out in practice. This could be for a technology reason (that app you thought you would design or introduce turns out to be more complex); a state capacity reason (maybe the government’s procurement rules make it impossible to do what you hoped, or the bureaucrats don’t have the right skills or right incentives to implement your intervention); or other bureaucratic and logistical reasons (e.g. your intervention aims to encourage firms to use digital platforms to advertise, but local financial rules make it difficult for those without credit cards to pay for these services, and a lack of documentation makes it hard for them to get credit cards). But too often programs are rolled out without testing that all these details can be resolved.
(c) Political economy constraints: one of the reasons that seemingly promising ideas have not already been implemented may be the presence of vested interests that mobilize to capture or prevent policies being rolled out. A year or two of testing may help understand where these bind, and help develop strategies for overcoming these constraints.
Examples of many of these problems arose in our attempts to evaluate matching grant programs, as detailed in this paper on learning from seven failed experiments in different African countries.
But it can be really hard to say give me money and let me tinker for a while
Chris’ suggestion to pilot a bunch of different things for a while and see what might have a chance of working, and what is definitely not working makes a lot of sense – and should be the way organizations roll out new policies. But as researchers this seems really hard to fund, and it also seems hard to make happen in the context of programs like a World Bank loan.
From the research side, most funders want a concrete research proposal for at most a 2-3 year horizon, with it being very clear how much budget you will need and what exactly you will do. The tinker and recalibrate approach instead would instead have you say “I’m going to try out these five or six ideas, will end up not doing most of them, might end up doing something completely different, and I can’t tell you what budget I’ll need for the intervention or what sample size I’ll need upfront – but it’s an important question and this is the right approach to addressing it, so please fund me” - this seems really hard to convince a funder to do – although perhaps it can be done through more funding of exploratory grants, at levels a bit higher than is currently done. The Stage 1 DIV grants, or some internal and external exploratory grants often allow for some piloting, but typically aren’t so happy for you to say you’ll try piloting many things at once.
From the World Bank loan side, similar issues arise – the process of negotiating these programs, deciding on which government ministries will be in charge, and the rules for implementation etc involves considerable negotiation and is very time-consuming – and then loans have results indicators tied to implementation of specified activities. We would need quite a large reform in this process to be able to say take the first 1-2 years to try out a bunch of different things, and then come back and finalize what will be done partway through. But I think there is a lot to recommend it.
In part what we need are examples of successful cases where this has been done. In the book, Chris suggests what is needed is a buzzword, and notes the growing success of “design thinking” as a philosophy, and how the IRC has been starting to use this approach with some projects. Having some well-documented cases of how both research projects and aid projects have managed to use this flexibility to tinker and test many ideas before finalizing what is done will be great as examples to foster more of this. Anyone got any great examples to share?
Join the Conversation