Oxfam

More DFIDistas on the blog: this time Nick York, DFID’s top evaluator and Caroline Hoy, who covers NGO evaluation, comment on Oxfam’s publication of a set of 26 warts-and-all programme effectiveness reviews.

Having seen Karl Hughes’s 3ie working paper on process tracing and talked to the team in Oxfam about evaluation approaches, Caroline Hoy (our lead on evaluation for NGOs) and I have been reading with considerable interest the set of papers that Jennie Richmond has shared with us on ‘Tackling the evaluation challenge – how do we know we are effective?’.

From DFID’s perspective, and now 2 years into the challenges of ‘embedding evaluation’ in a serious way into our own work, we know how difficult it often is to find reliable methods to identify what works and measure impact for complex development interventions.  Although it is relatively well understood how to apply standard techniques in some areas – such as health, social protection, water and sanitation and microfinance – there are whole swathes of development where we need to be quite innovative and creative in finding approaches to evaluation that can deal with the complexity of the issues and the nature of the programmes.  Many of these areas are where NGOs such as Oxfam do their best work.

Guest post from ace evaluator Dr Karl Hughes (right, in the field. Literally.)

Just over a year ago now, I wrote a blog featured on FP2P – Can we demonstrate effectiveness without bankrupting our NGO and/or becoming a randomista? – about Oxfam’s attempt to up its game in understanding and demonstrating its effectiveness.  Here, I outlined our ambitious plan of ‘randomly selecting and then evaluating, using relatively rigorous methods by NGO standards, 40-ish mature interventions in various thematic areas’.  We have dubbed these ‘effectiveness reviews’.  Given that most NGOs are currently grappling with how to credibly demonstrate their effectiveness, our ‘global experiment’ has grabbed the attention of some eminent bloggers (see William Savedoff’s post for a recent example).  Now I’m back with an update.

A while back, I wrote about some amazing Oxfam women I met in East Africa. Here’s another, this time from the Philippines.

Lan (real name Lilian, but Filipinos never use real names) is one of those quiet but effective (and very determined, and maybe not so quiet….) women that abound in development work. She was formerly our country director in the Philippines, but has now moved to head up a project on ASEAN (more on that below). She is also yet another Oxfam woman with a remarkable story. In 1988, as a 28 year old Communist Party activist in the Philippines civil war, her own Party denounced and arrested her on trumped-up charges of being involved in an intra-Party assassination. They held her for 6 months in the mountains, blindfolded and handcuffed in a cage. She and the other prisoners were tortured physically, mentally and emotionally. At least she avoided the fate of prisoners in other camps, who were forced to play ‘eeny meeny miny mo’, with the loser taken out, killed, and their blood smeared over the remaining prisoners.

Last week I visited Oxfam’s Philippines programme. Such trips follow a pretty standard format - our national staff and relevant partners with the moringa farmers whisk me through a series of site visits and conversations with farmers, civil society organizations, local government officials and anyone else who’ll talk to you. For a few days, I’m engrossed, wrestling on multiple levels, first to understand the intricacies of the projects, and then to try and get at the meta-questions: what are the strengths and weaknesses in our work? What could we be doing better? Is there a clear power analysis and theory of change? Discussions continue in vehicles to and from the visits, over dinner and (sometimes) in the bar, as everyone grapples with the incredibly difficult business of ‘doing development’. It’s intense and definitely the best bit of the job.

I went to Mindanao, one of the poorest and most conflict-ridden islands in the Philippines archipelago, and home to 23m of its 94m population. The focus was our livelihoods work (I hate the term, but can’t think of anything better to describe the complex ways poor people find to put food on the family table). Such work forms the backbone of many of Oxfam’s programmes. In Mindanao, we’re working with women farmers to introduce new crops or upgrade existing ones:

I’ve been reading the set of papers Oxfam recently published on local governance and community action (see previous blog) and was struck by how central the issue of ‘implementation gaps’ is in our work.

An implementation gap is where a set of institutions (often created via decentralization), policies or budgets (or all three) exist on paper, but are absent on the ground. Such a situation provides a particularly good entry point for an INGO like Oxfam because it reduces political risk (you are supporting the implementation of what the state has already agreed) and the benefits are likely to be easier to achieve and can have a galvanizing effect – plucking low-hanging fruit is great for morale and motivation. In terms of power analysis, this is about making the most of ‘invited spaces’ rather than creating new ones.

I recently gave a talk about ICT and Development at the annual Re:Campaign conference in Berlin, organized by Oxfam Germany. Anyone who knows me will realize that this is a bit odd – despite being a blogaholic, I am actually Rubbish At Technology. In front of 300 trendy, young (sigh) i-thingy wielding activists, I felt like a Neanderthal at a cocktail party. Still, at least the fear of being shamed up finally got me tweeting two weeks before the conference.

I decided to make a virtue of necessity and set out some core processes in development, and then reflected on what ICT does/doesn’t contribute. Why take this approach (apart from being a techno-caveman, that is)? Because there’s too much magic bulletism in development –microfinance, GM crops and now ‘cyber utopianism’. What all of these have in common is that they are too often presented as ‘get out of jail free’ cards, delivering development without all the messy business of politics and struggle. At best, new technologies shift power balances, sometimes favourably, sometimes not, but they don’t replace the process of struggle in development.