Results

I’m at one of those moments where all conversations seem to link to each other, I see complex systems everywhere, and I’m wondering whether I’m starting to lose my marbles. Happily, lots of other people seem to be suffering from the same condition, and a bunch of us met up earlier this week with Matt Andrews, who was in the UK to promote his fab new book Limits to Institutional Reform in Development (I  rave reviewed it here). The conversation was held under Chatham House rules, so no names, no institutions etc.

Whether you work on complex systems or governance reform or fragile states, the emerging common ground seems to be around what not to do and to a lesser extent, the ‘so whats’. What can outsiders do to contribute to change in complex, unpredictable situations where, whether due to domestic opposition or sheer irrelevance to actual context, imported blueprints and ‘best practice guidelines’ are unlikely to get anywhere?

In his book Matt boils down his considerable experience at the World Bank and Harvard into a proposal for ‘PDIA’ – Problem Driven iterative adaptation, which I described pretty fully in my review. The conversation this week fleshed out that approach and added some interesting new angles.

I’m consistently astonished by how little we know about the important stuff in development. Take the Millennium Development Goals – the basis for innumerable aid debates, campaigns, and negotiations. A large chunk of the MDG agenda concerns the size and quality of public spending – on health, education, water, sanitation etc. So obviously, the first thing we need is to know how much governments are spending on these things, right?

Well no actually, because we don’t have those numbers. Until now. Oxfam has teamed up with an influential and well-connected NGO, Development Finance International, which advises developing country governments around the world. Working with a network of government officials, DFI has pulled together and analysed the budgets of 52 low and middle income countries (With another 34 to follow). The result is a new database, called Government Spending Watch, (summary of overall project here) and a report ‘Progress at Risk’, previewed in Washington last Friday in a joint DFI/Oxfam America event to coincide with the IMF and World Bank Spring meetings. The full report won’t be ready ‘til May, but an initial draft exec sum is available, and here’s what it says.

The debate on evidence and results continues to rage. Rosalind Eyben (left) and Chris Roche (right, dressed for battle), two of the organisers of April’s Big Push Forward conference on the Politics of  Evidence, kick off a discussion. Tomorrow Chris Whitty, DFID’s Director of Research and Evidence and Chief Scientific Adviser, and Stefan Dercon, its Chief Economist, respond.

Distinct from its more general usage of what is observed or experienced, ‘evidence’ has acquired a particular meaning relating to proof about ‘what works’, particularly through robust evidence from rigorous experimental trials. But no-one really believes that it is feasible for external development assistance to consist purely of ‘technical’ interventions. Most development workers do not see themselves as scientists in a laboratory, but more as reflective practitioners seeking to learn how to support locally generated transformative processes for greater equity and social justice. Where have these experimental approaches come from and what is at stake?

Recently I had a lot of conversations about evidence. First, one of the periodic retreats of Oxfam senior managers reviewed our work on livelihoods, humanitarian partnership and gender rights. The talk combined some quantitative work (for example the findings of our new ‘effectiveness reviews’), case studies, and the accumulated wisdom of our big cheeses. But the tacit hierarchy of these different kinds of knowledge worried me – anything with a number attached had a privileged position, however partial the number or questionable the process for arriving at it. In contrast, decades of experience were not even credited as ‘evidence’, but often written off as ‘opinion’. It felt like we were in danger of discounting our richest source of insight – gut feeling.

In this state of discomfort, I went off for lunch with Lant Pritchett (right – he seems to have forgiven me for my screw-up of a couple of years ago). He’s a brilliant and original thinker and speaker on any number of development issues, but I was most struck by the vehemence of his critique of the RCT randomistas and the quest for experimental certainty. Don’t get me (or him) wrong, he thinks the results agenda is crucial in ‘moving from an input orientation to a performance orientation’ and set out his views as long ago as 2002 in a paper called ‘It pays to be ignorant’, but he sees the current emphasis on RCTs as an example of the failings of ‘thin accountability’ compared to the thick version.