Each month People, Spaces, Deliberation shares the blog post that generated the most interest and discussion. In November 2015, the featured blog post is "If using ‘Theories of Change’ cannot transform the way you operate, why bother? " by Suvojit Chattopadhyay .
Recently, Craig Valters published new work on theories of change. He calls not for a new tool (product) but for a more careful approach (process) to practicing and engaging in development. That is, changing the state of the world for someone. And learning from it. And, ideally, communicating that learning. (Craig is pessimistic that we are near actually ushering in a learning agenda to replace the results agenda. On this, I hope he is wrong.)
In this post, I echo and expand on the idea of theories of change as allowing “space for critical reflection” (p. 4) and push back slightly on two of the outlined ‘key principles’ of a theory of change approach: being ‘locally led’ and thinking ‘compass, not map.’ I also include some of the comments Craig made on the original version of this post, here.
I have two disclaimers, given points raised both in the paper and in Suvojit’s follow-up blog. The first is a musing, though I have adopted the theory of change language along with the herd. I wish we could still revise it to hypotheses of change or ideas of change or stuff that might matter because we thought hard about it, looked at what had been done before, and talked to people about what could be done now (or something else catchier but far more tentative, humble, and open to updating than 'theory'). Alas. On the brighter side, Craig notes that, at least, “theory implies we have to think really hard about it, even if what we end up with is not a theory in the social science sense of the word.”
The second is a confession. I really like boxes and arrows— not as the definitive product associated with a theory of change but as some means of organizing ideas that people can stand around, look at, point to, and ask, “have we learned anything about how this arrow really works?” While I don’t want to foist the need for a visual on anyone, especially if it is just going to end as a bad flowchart, I feel I should at least lightly advocate that a visual can be a useful tool for learning and may be friendlier to revisit than a lengthy narrative, and it's usually prettier. In his follow-up responses, Craig echoes that a diagram, no matter how pretty, “is not in itself a Theory of Change.” I concur.
How useful is 'focusing on results' for development work? It may make an organization more cost-efficient but not necessarily more effective as it is usually unrealistic, time-consuming and misleading.
How do donors aim for “results” without setting up a counterbureaucracy that disrupts rather than encourages good development programs?
A recent Independent Commission for Aid Impact report has taken the U.K. Department for International Development to task for doing just that, which in turn demands a serious reconsideration of how DfID thinks about results and accountability.
Of course, these critiques are hardly new. However this isn’t another nongovernmental organization or academic report slating the “results agenda,” but an independent body that has specifically been set up to ensure the effectiveness of aid and — based on 44 previous reports — is providing evidence about how the results agenda unfolds in practice.
In a nutshell, ICAI argues that DfID today knows better than ever before when and where taxpayers’ money is being spent, but not what that spending actually achieves. ICAI found that the results agenda has tended to prioritize short-term economy and efficiency over long-term, sustainable impact. It has brought “greater discipline” and “greater accountability for the delivery of aid” but also a focus on quantity of results over quality.
Not everything ICAI has to say is bad news; but most of it is. The ICAI findings undoubtedly hold broader relevance for other donors who are taking a similar approach to their result agenda.
Over the course of a year, each Leadership Fellow develops an Action Plan for reform back home, ranging from girls’ education to police training to civil society strengthening, and will work on it during their UK visit, where they get inputs from people like me, discussions and visits to the UK Parliament and elsewhere.
I was presenting on theories of change (ToCs) – here’s my powerpoint. My co-presenter (from a UK thinktank) defined a ToC as ‘a conceptual map of how activities lead to outcomes’. As you might imagine, I disagreed with the implied linearity of that. But the disagreement, and the views of those present was interesting.
I have been thinking a lot about ‘theories of change’ this week (as I was here). Actually, I have been thinking more about ‘conceptual models,’ which was the term by which I was first introduced to the general idea* and the term I still prefer because it implies more uncertainty and greater scope for tinkering than does ‘theory.’ (I accept that ‘theory of change’ has been branded and that I have to live with it, but I don’t have to like it.)
Regardless of the term, the approach of thinking seriously about how behavioral, social and economic change happens is important but often overlooked during the planning stages of projects/programs/policies and linked evaluations. Moreover, they are glossed over in the analysis and reporting stages, left to academic speculation in the discussion section of an evaluation paper and not informed by talking systematically to those people who were intended to benefit from the program.
I think there is growing recognition that building a theory of change is something that should happen, at least in part, backwards (among other places where this is discussed is in ‘evidence-based policy’ with the idea of a ‘pre-mortem‘ and ‘thinking step-by-step and thinking backwards‘). That is, you start with the end goal, usually some variant of ‘peace,’ ‘satisfaction,’ ‘wellbeing,’ ‘capabilities,’** etc., in mind and work backwards as to how you are going to get there from here.
If you are interested in Theories of Change (ToCs), you have to read Craig Valters’ new paper ‘Theories of Change in International Development: Communication, Learning or Accountability’ or at least, his accompanying blog. The paper draws on the fascinating collaboration between the LSE and The Asia Foundation, in which TAF gave LSE researchers access to its country programmes and asked them to study their use of ToCs. That means Craig has been able to observe their use (and abuse) in practice.
What this paper helps answer is the question I raised a while ago – will ToCs go the way of the logframe, starting out as a good idea, but being steadily dumbed down into a counterproductive tickbox exercise by the procedural demands of the aid business?
Certain countries seem to produce more than their share of great programmes. Vietnam is one, and Tanzania appears to be another. After the much-blogged-on Twaweza workshop in Tanzania last week, I headed up North to visit the Chukua Hatua accountability programme. It’s one of my favourites among Oxfam’s governance work, not least because it has a really top notch theory of change (keep clicking) I often get asked for a good real life practical example of a ToC – in governance work, this is the best I’ve seen.
Over a series of conversations with Oxfam staff and partners, village activists, officials and others, one intriguing issue struck me: even if you start out as innovative, what happens next?
Let me explain. Chukua Hatua started out with a really interesting theory of change – adopt an evolutionary approach of variation-selection-amplification. That meant trying out lots of things in phase 1 (2010/11), then sifting through the results to identify the most successful variant(s) and scaling that up.
The variant that stood out was that of animation: training farmers selected by their communities to become animators – entrepreneurial, networked activists identifying problems in their communities and bringing people together (both villagers and those in power) to find solutions. This has worked brilliantly, so phase 2 (2012/13) has scaled that up.
Twaweza’s Varja Lipovsek, (Learning, Monitoring & Evaluation Manager) and Rakesh Rajani (Head), respond to this week’s
series of posts on their organization’s big rethink.
That Duncan Green dedicated three posts on Twaweza’s ‘strategic pivot’ may signal that our work and theory of change are in real trouble, but we prefer to take it as a sign that these issues are of interest to many people working on transparency, accountability and citizen-driven change. His posts follow a terrific two day evaluation meeting. Here are a few clarifications and takeaways.
Spiritual matters first. We very much believe that Twaweza’s soul remains intact: we want to contribute towards change in complex systems in East Africa, by promoting and enabling citizens to be active agents and shape their lives. Our experience over the past four years has made us question much of how we ‘do’ citizen agency, but we are not quite throwing out the baby with the bathwater.
For example, in our original approach we didn’t want to be prescriptive about citizen action; we wanted to expand choices and leave it up to people to decide, what we called an ‘open architecture’ approach to social change. Sounds good; problem is that it doesn’t work so well in practice and the evidence of successful change suggests a need for less openness and more focus. New evidence about the bandwidth that poor people have to make good decisions provides useful insights on what one can realistically expect people to do.
This is the last in a series of three posts on Twaweza, a fascinating NGO doing some pioneering work on accountability in East Africa, whose big navel gaze I attended last week. Post one covered Twaweza’s theory of change and initial evaluation results; yesterday I got onto the critique of its thinking and action to date. Today I’m digging deeper into some of the underlying issues.
Given its rethink, Twaweza is now contemplating a shift in direction – while keeping its focus on citizen agency, focus in on education (rather than try and cover education, health and water); reduce the number of partners; do more things on its own (eg research or education programming); expand successful areas such as policy and advocacy; do more experiments to uncover what works and help the organization ‘fail faster’ and so move on to new stuff.
Plenty of good ideas in there, but it also seems to me to mark an intellectual retreat from the initial commitment to finding new ways to achieve change in complex systems. I think there’s a strong case for digging deeper into complexity, rather than retreating from it. One suggestion that moves in the right direction is to set up a ‘positive deviance lab’, dedicated to detecting and then understanding examples of success in citizens’ action across East Africa.
Yesterday I sketched out the theory of change and initial findings on the first four years of work by an extraordinary East African NGO, Twaweza. Today I’ll move on to what some NGO people (but thankfully no-one in Dar es Salaam last week) insist on calling ‘the learnings’ about the flaws and gaps in its original theory of change (described in yesterday’s post).
First, there’s a big ‘black box’ containing Twaweza’s rather large assumption that giving people information (eg about failing education systems), would lead to them taking action to change things. What issues in the black box determine whether this is true or not?
Evan Lieberman (one of Twaweza’s many evaluators, from Princeton University) called this the ’secret sauce’ – the miracle that links information to action. His team had come up with a smart attempt to identify some of the sauce’s ingredients – conditions for a →b:
Do I understand the info? →Is it new info? →Do I care? →Do I think that it is my responsibility to do something about it? →Do I have the skills to make a difference? →Do I have the sense of efficacy to think that my efforts will have an impact? →Are the kinds of actions I am inspired to take different from what I am already doing? →Do I believe my own individual action will have an impact? →Do I expect fellow community members to join me in taking action? Evan argued that only if the answer to all of these is yes, will the black box indeed turn information into action.
Actually it’s worse than that – they missed some pretty big ones (‘do I have the time to do this, on top of everything else?’ ‘Will I run any personal risks if I do this?’). It’s a hell of an intimidating set of conditions and, as was pointed out, the danger is that accountability proponents will just latch onto one of the steps, then wonder why nothing is popping out at the outcome end.