I was lucky to recently attend a workshop on justice and governance impact evaluations in the wonderful city of Istanbul. The spark of the workshop discussions lived up to the liveliness of the location.
Some time ago I blogged about the pros and cons of impact evaluations for justice projects in developing countries. Since then, interest in impact evaluations in the justice sector has grown at the World Bank and within the larger development community.
The World Bank now has a nascent portfolio of justice sector impact evaluations. Four are underway - in Senegal, Colombia, Kenya and Jordan - with six more in the early stages of development in the Caribbean and Central and South Asia. In addition to covering most regions of the globe, the topic areas of these impact evaluations span many of the core interests of justice policy makers and practitioners globally: improving the performance of judges and magistrates; measuring the impact of new court infrastructure; targeting the delivery of legal aid to the poor; and programs to prevent youth involvement in crime.
The World Bank’s Governance Global Practice and Development Impact Evaluation program are currently working on a white paper which will lay out some guiding principles for designing and undertaking justice impact evaluations in developing countries, including pitfalls to avoid. Given their method, impact evaluations are useful in narrowing down whether reform X causes outcome Y, but the very nature of the method also means they are only suitable for answering a sub-set of the many questions of interest to justice policy makers. The Istanbul workshop involved matching up these ‘questions of interest’ from different project teams with academics schooled in impact evaluation methods.
A common critique of impact evaluations is that they are mainly useful in answering ‘small’ reform questions. Some argued at the workshop that given the relative paucity of empirical information about what works in justice and governance, perhaps we should be focusing on finding ‘fuzzier’ answers to bigger-picture questions - rather than gathering robust evidence on smaller ones. I have some sympathy for this view.
On the other hand, when it comes to reform I’m somewhat of an incrementalist. Absent a natural disaster, war or other large shock, most reform ends up being the agglomeration of a myriad of smaller changes. It so happens that these incremental reforms are more likely to be suited to testing by way of an impact evaluation – and we can see whether each step is in the right direction.
Whichever side of the big/small divide you fall on, it’s important to keep in mind that the purported power of impact evaluations stems from an assumption that the rigorousness of the data produced is compelling to decision makers.
We know from nearly all contexts that data is usually only one factor that influences policy makers – and the rigor of that data is only one factor in its take up. It is therefore useful, before embarking on the long road of an impact evaluation, to take the temperature of the policy environment to assess whether this kind of evidence will be compelling to decision makers.
That said, in addition to testing the substantive merit of different reforms, impact evaluations can have potentially important “spillover benefits”. They can support the growth of a culture of data collection and decision making founded on evidence. Over the past 18 months in Kenya, as part of preparations for an impact evaluation, we have been supporting great work already underway by the Judiciary’s Performance Management Directorate.
This work has sought to better understand existing case loads, sites of blockages and update the Judiciary’s administrative data collection system. All of this work has useful spillovers for the Kenyan Judiciary – whatever the outcome of the ultimate impact evaluation.
No doubt we will learn more important lessons on the usefulness and feasibility of impact evaluations as the field continues to grow. We hope to encourage continued conversation on this topic with the production and dissemination of our white paper later this year.
What do you think? Tell us in the comments.
Join the Conversation