I just finished reading a recent working paper by Martin Williams, titled “External validity and policy adaptation: From impact evaluation to policy design”.
In this paper, Martin tackles the question – how will a policymaker apply evidence available to them to design a policy/programme that will fix a particular problem at hand? He first takes us through the ways in which we think of this currently – primarily by attempts to strengthen the external validity of evaluations – and points out the limitations of these approaches. The central critique is that most of this thinking puts the evaluators/researchers at the centre and tries to devise ways in which the evidence generated by their research can be generalised beyond their specific study samples. This is at odds with what a policymaker (in this paper, a public official in a given country) needs in order to make decisions about how to use evidence from elsewhere to design a policy/programme for their specific context.
The answer, Martin suggests, is ‘mechanism mapping’ – a five-step process where the public official lays out:
Step 1: The theory of change (ToC) of the programme that generated the evidence at hand;
Step 2: The context within which the programme ToC worked;
Step 3: The context at the destination – local factors that will affect a potential replication;
Step 4: By doing the above, interrogate the evidence with the new context, and suggest design modifications; and
Step 5: Iterate, until the gaps are plugged.
Sounds reasonably straight-forward, and would seem to function quite well as a way to break it down for a public official who needs inputs into designing programmes. At the same time, introducing a structured way to do this lends a degree of rigour that will benefit the process.
A point made repeatedly in this paper is that a lot of the thinking around research, evaluation and evidence is focused on the researchers/evaluators. As a result, many of these conversations focus on whether (and how) a certain body of evidence that has been generated would apply ‘elsewhere’, without necessarily specifying a destination for this policy design. Thinking about this from the perspective of the public official ensures that the problem is framed as “what tools do public officials need to apply evidence to their specific context” instead of “how can our evidence be applied in the design of public policy elsewhere”. This switch in perspective allows us to do more than just empathise with the constraints that public officials function within. It enables us to think of solutions that focus on the end-user, even if that might mean trade-offs in statistical rigour.
This brings me to an important factor in how this would work – is the availability of information, and the institutional set-up where the mechanism mapping might take place. How well equipped is a public official in a capital city of a developing country to do this? Who should they consult? Will they? How decentralised is policymaking in the said country, and how does that factor in? My first frame of reference is India, so please pardon my concern for size and complexity. It will ultimately boil down to having reasonable limits to such consultation, and to the amount of information that will be fed into such an exercise.
Another factor to consider is the capacity of public officials. At a time when even researchers hardly fully understand (or bother with) complex theories of change, assuming that public officials will do this on their own requires a certain faith in their capacity. Martin’s core experience of bureaucracy and public management comes from Ghana, where he spent two years working within the Ministry of Trade and Industry. This is reflected in what I think is a key characteristic of this paper – Martin’s ability to think like a public official. Working in a stable state with a reasonably functional public sector, as opposed to a nascent or a failing state, shapes one’s perspectives of state and bureaucratic capacity in significant ways. If your canvas is the former, you are more likely to trust that public officials are reasonably able to competently design and implement policies, and that their work lies within a stable policy context. But even so, public officials will need training and support. And researchers will need to be trained to resist the temptation to turn up their noses at these inevitably ‘messy’ exercises.
Below, is an illustration (this is not from the paper) of how a public official might approach this process, beyond going through the ‘mechanism mapping exercise’ – by considering the combinations thrown up by the interaction between “complexity of the ToC” and “complexity of Context”. For simplicity, I present four scenarios, and in each, the responses a public official might have in order to move ahead with policy design. The top-right quadrant represents a complex programme ToC that needs to be applied to a complex context – a big challenge, where the response has to be a willingness to experiment and iterate. On the other hand, is the quadrant on the bottom-left, which is likely to be satisfied by technical fixes. Note that this figure does not directly mention ‘information gaps’, although a high level of complexity indicates a high probability of information asymmetries on both counts – in the ToC, as well as in the Context.
Finally, I did not forget to ask myself this – “did we not already know all this?” Like with a lot of the work on ‘adaptive management’ and PDIA, I think the honest answer is that we do, but certainly not enough of it, and not in a systematic manner. Good programmes travel, and policy adaptation is a continuous process – public officials (and other practitioners) learn and adapt continuously. ‘Mechanism mapping’ is a powerful tool to systematise this iterative learning process, and potentially, shorten the learning curve. In a context where the application of evidence to policy design cannot be taken for granted, this paper and its illustration of ‘mechanism mapping’ is an important contribution.
Here is the policy memo. What we need next is a training module for public officials – I hear its coming soon, to a website near you…