Syndicate content

Decision Making

What does it mean to do policy-relevant research and evaluation?

Heather Lanthorn's picture

Center for International Forestry Research (CIFOR) researchers upload the data to see the resultsWhat does it mean to do policy-relevant research and evaluation? How does it differ from policy adjacent research and evaluation? Heather Lanthorn explores these questions and offers some food for thought on intention and decision making.

This post is really a conversation with myself, which I started here, but I would be happy if everyone was conversing on it a bit more: what does it mean to do research that is ‘policy relevant’? From my vantage point in impact evaluation and applied political-economy and stakeholder analyses, ‘policy relevant’ is a glossy label that a researcher or organization can apply to his/her own work at his/her own discretion. This is confusing, slightly unsettling, and probably dulls some of the gloss off the label.

The main thrust of the discussion is this: we (researchers, donors, folks who have generally bought-into the goal of evidence- and evaluation-informed decision-making) should be clear (and more humble) about what is meant by ‘policy relevant’ research and evaluation. I don’t have an answer to this, but I try to lay out some of the key facets, below.
 
Overall, we need more thought and clarity – as well as humility – around what it means to be doing policy-relevant work. As a start, we may try to distinguish work that is ‘policy adjacent’ (done on a policy) from work that is either ‘decision-relevant’ or ‘policymaker-relevant’ (similar to ‘decision-relevant,’ (done with the explicit, ex ante purpose of informing a policy or practice decision and therefore an intent to be actionable).
 
I believe the distinction I am trying to draw echoes what Tom Pepinsky wrestled with when he blogged that it was the “murky and quirky” questions and research (a delightful turn of phrase that Tom borrowed from Don Emmerson) “that actually influence how they [policymakers / stakeholders] make decisions” in each of their own idiosyncratic settings. These questions may be narrow, operational, and linked to a middle-range or program theory (of change) when compared to a grander, paradigmatic question.
 
Throughout, my claim is not that one type of work is more important or that one type will always inform better decision-making. I am, however, asking that, as “policy-relevant” becomes an increasingly popular buzzword, we pause and think about what it means.

Quote of the Week: Arnab Goswami

Sina Odugbemi's picture

"I don't believe in creating an artificial consensus, which some people are comfortable with. So if there is something wrong, you can ask yourself two questions: 'Why did it happen? Will the people who did it go unpunished?' "
 

Arnab Goswami, an Indian journalist and the editor-in-chief of Indian news channel Times Now. He anchors The Newshour, a live debate show that airs weekdays on Times Now and hosts the television programme Frankly Speaking with Arnab.
 

Buffet of Champions: What Kind Do We Need for Impact Evaluations and Policy?

Heather Lanthorn's picture
I realize that the thesis of “we may need a new kind of champion” sounds like a rather anemic pitch for Guardians of the Galaxy. Moreover, it may lead to inflated hopes that I am going to propose that dance-offs be used more often to decide policy questions. While I don’t necessarily deny that this is a fantastic idea (and would certainly boost c-span viewership), I want to quickly dash hopes that this is the main premise of this post. Rather, I am curious why “we” believe that policy champions will be keen on promoting and using impact evaluation (and subsequent evidence syntheses of these) and to suggest that another range of actors, which I call “evidence” and “issue” champions may be more natural allies. There has been a recurring storyline in recent literature and musings on (impact) evaluation and policy- or decision-making:
  • First, the aspiration: the general desire of researchers (and others) to see more evidence used in decision-making (let’s say both judgment and learning) related to aid and development so that scarce resources are allocated more wisely and/or so that more resources are brought to bear on the problem.
  • Second, the dashed hopes: the realization that data and evidence currently play a limited role in decision-making (see, for example, the report, “What is the evidence on evidence-informed policy-making”, as well as here).
  • Third, the new hope: the recognition that “policy champions” (also “policy entrepreneurs” and “policy opportunists”) may be a bridge between the two.
  • Fourth, the new plan of attack: bring “policy champions” and other stakeholders in to the research process much earlier in order to get up-take of evaluation results into the debates and decisions. This even includes bringing policy champions (say, bureaucrats) on as research PIs.

There seems to be a sleight of hand at work in the above formulation, and it is somewhat worrying in terms of equipoise and the possible use of the range of results that can emerge from an impact evaluation study. Said another way, it seems potentially at odds with the idea that the answer to an evaluation is unknown at the start of the evaluation.

‘Relevant Reasons’ in Decision-Making (3 of 3)

Heather Lanthorn's picture

This is the third in our series of posts on evidence and decision-making; also posted on Heather’s blog. Here are Part 1 and Part 2
***
In our last post, we wrote about factors – evidence and otherwise – influencing decision-making about development programmes. To do so, we have considered the premise of an agency deciding whether to continue or scale a given programme after piloting it and including an accompanying evaluation commissioned explicitly to inform that decision. This is a potential ‘ideal case’ of evidence-informed decision-making. Yet, the role of evidence in informing decisions is often unclear in practice.

What is clear is that transparent parameters for making decisions about how to allocate resources following a pilot may improve the legitimacy of those decisions. We have started, and continue in this post, to explore whether decision-making deliberations can be shaped ex ante so that, regardless of the outcome, stakeholders feel it was arrived at fairly. Such pre-commitment to the process of deliberation could carve out a specific role for evidence in decision-making. Clarifying the role of evidence would inform what types of questions decision-makers need answered and with what kinds of data, as we discussed here.

How Can a Post-2015 Agreement Drive Real Change? Please Read and Comment on this Draft Paper

Duncan Green's picture

The post-2015 discussion on what should succeed the Millennium Development Goals (MDGs) is picking up steam, with barely a day going by without some new paper, consultation or high level meeting. So I, along with Stephen Hale and Matthew Lockwood, have decided to add to the growing slush-pile with a new discussion paper. We want you to read the draft (see right) and help us improve it. Contributions by 5 November please, either as comments on the blog, or emailed to research[at]oxfam.org.uk.

The paper argues that there’s an urgent need to bring power and politics into the centre of the post-2015 discussion. To have impact, any post-2015 arrangement has to take into account the lessons of over a decade of implementing the existing MDGs, and be shaped by the profound global change since the MDGs were debated over the course of the 1990s and early noughties.  We’re hoping that this will be at the centre of this week’s discussions in London linked to the High Level Panel and in Berlin at the Berlin Civil Society Center on Development post 2015.

I Only Hear What I Want to Hear – And So Do You

Anne-Katrin Arnold's picture

Do you know these people who always only hear what they want to hear? Who interpret everything in a way that fits their own views? Yes? Deal with them every day? Well, chances are, you’re one of them. Biased information processing is a common phenomenon. It happens when the information we receive is out of sync with what we believe to be true, or want to be true, or when the information is inconvenient for us. This obviously has huge implications for communication campaigns in development.