Syndicate content

Will Midterm Evaluations Become the Dinosaurs of Development?

Milica Begovic's picture

I argued a few months back that information we get from story-telling is fundamentally different to what we get from polls and surveys. If we can’t predict what’s coming next, then we have to continuously work to understand what has and is happening today. (See: Patterns of voices from the Balkans – working with UNDP)

Methods we’re all used to using (surveys, mid-term evaluations) are ill prepared to do that for us and increasingly act as our blindfolds.

Why stories?

As I started working through the stories we collected, this question has become even stronger.

To give you some background, we started testing whether stories could help us:

  • Gain insights into dynamics of local life in national parks
  • Monitor integration of Roma into the broader community
  • Assess to what extent UNDP corporate rules and procedures enable staff to react efficiently and timely in daily work
  • Identify what factors facilitate or hinder citizen engagement with public administration
  • Understand to what extent communities living in Chernobyl face discrimination

Why are we doing this? Continually collecting stories allows us to identify patterns and trends, which can help us to:
  • Detect weak signals: We’d like to be able to monitor and react timely to shifting attitudes, subtle behaviour changes or hardening of beliefs – whether it is distrust in local government or increased fear of prosecution. This is hard to achieve through traditional surveys.
  • From linear to iterative planning: We’d like to be able to send parts of new plans and strategies out for testing with different segments of communities who continually collect stories, directly feeding their input into the design of our work. Those who tell stories become sensors who can provide rapid feedback or help us to detect early warning signs.
  • Cost saving: We’d like to be able to understand better – and quicker – what works and what doesn’t in our projects – whether it’s a campaign, a new law or a service, so that we can avoid cases where well-meaning policies could hurt communities we work with.
  • More effective communication: Tracking trends in real time can move us beyond how many people we’ve reached or how many ‘likes’ or ‘retweets’ we had to what is the extent of perception changes we’re seeing as a result of various campaigns.  Also, we’d like to be able to advocate for change using anecdotes from communities we work with (as opposed to building a corporate storyline, largely insulated from those who it is meant for in the first place).
  • Build foresight into today’s decisions:  We’d like the decisions we make today to be informed by considering several possible future scenarios based on new trends – whether it is a perception of importance of the informal economic sector or attitudes of different ethnic groups to instability. Also, if we can help shape dialogue on critical issues by asking ourselves questions like: how do we get communities to share more stories like this? (see chart below)

Reading this graph is a bit counter-intuitive, the weight of answers is weak to strong across the page. So, overwhelming a majority of citizens interpreted their experiences in dealing with public administration as instances where they had very little to no control over decision making (and among respondents, it is mainly women). In our future work, we’d like to see the experiences shift toward the right side of the page, and we may be particularly concerned about women feeling little control in decision making.

This all started as an experiment for us, but having seen how it works, it will be difficult to go back to the way we’ve been doing things.

The question now is: how do we get better and faster at using this method far and wide in order to invest funding more effectively and do our development work better?

And while we ponder this question for a bit, my next post will address the most often heard critique of storytelling I have heard over the last few months.

This post first appeard on the Eurasia blog
Photo Credit: Dominic Chavez / World Bank
Follow PublicSphereWB on Twitter

Add new comment