As part of a new series looking how institutions are approaching impact evaluation, DI virtually sat down with Nick York, Head of Evaluation and Gail Marzetti, Deputy Head, Research and Evidence Division. For Part I of this series, see yesterday’s post. Today we focus on DFID’s funding for research and impact evaluation.
Development Impact (DI): DFID is probably funding more development research than any bilateral aid agency at the moment. Can you tell us a bit about your funding levels for research over the last couple of years, and where the main opportunities are for researchers wanting to apply for this funding?
Gail Marzetti: Yes, Ministers and the Management Board of DFID have over the last three years recognised that increasing investment in high quality research and evidence will be key to delivering development results. The research budget has been increasing over the last few year and was £222 million in the last (2011/12) financial year.
Research programmes that are commissioned directly by DFID are generally awarded through open competition, and go through a peer review process. This is usually a two stage process. In the first stage, short expressions of interest are received and reviewed by a group of internal reviewers, with a range of relevant expertise. Those shortlisted at this stage are then invited to submit full proposals. Full proposals are evaluated by a panel of reviewers that includes at least one independent external reviewer. Evaluation criteria are published at the time of calls for proposals. Those wanting to apply for DFID Research Funding can register their interest on our website ‘Research for development’ found at; www.dfid.gov.uk/R4D to receive alerts when we issue calls for research. DFID also commissions research through indirect partnering with other organisations such as the UK Research Councils who manage a competitive process of research calls on our behalf.
DI: How do you determine what areas to focus on? What is the main objective you are trying to maximize with funding allocations – general human knowledge, country-specific policy knowledge, short-run improvements in well-being, or something else?
GM: DFID’s Research and Evidence Division’s investment in research is primarily focused on the production of global public goods, country office investment in research is primarily focused on research questions relating to a country or regional context.
DFID’s Research and Evidence Division has 3 broad aims in its research:
- to support the development of new technologies which would impact on poverty or the effects of poverty, such as new drugs, vaccines or drought-tolerant crops;
- to find better and more cost-effective ways of delivering aid and development assistance to those who need it; and
- to increase understanding of key development questions to support best policy choices
We have 7 broad thematic priority areas: agriculture; climate and environment; governance, conflict and social development; growth; health and education; short term evidence on impact and innovation; and research into use.
DI: Where do you see your priorities or balance between funding efficacy trials (e.g. a particular intervention or policy works under ideal conditions on a small scale) versus effectiveness trials (e.g. a particular intervention works under real-world conditions with typical messy implementation) versus mechanism experiments (e.g. testing particular mechanisms through which a policy is supposed to work when testing the policy itself may be difficult)?
Nick York: To some extent we are interested in all 3, but particularly the translation of research into implementation in the real world and this is where evaluation comes in. Sometimes the reason an intervention does not work is quite simply because it has not actually been implemented or is poorly targeted, as is often shown in baseline surveys. Recently, a cash transfer programme in Ghana was redesigned before implementation because the scoping work for the baseline survey had shown that the level of the cash transfer was too small to be likely to make much of a difference.
DI: While researchers are very, very appreciative of the funding DFID has been putting into impact evaluations, there has been a lot of concern about the restriction to only “low-income” countries. There are several reasons for this concern: i) we are moving to a situation where a majority of the world’s poor population are going to be living in middle-income countries; ii) the relative value of knowledge generation may well be higher for countries which are closer to the world frontier than for the poorest countries where there are a lot of basics that just need to be done right; and iii) a country-focus may work against a focus on interesting questions or policies – e.g. the only chance to evaluate a policy designed to improve firm productivity may happen in a relatively large middle income country where there are sufficient firms to enable the researcher to have enough power to estimate its effects, and where the government is actually willing to enact such a policy. We understand there are likely political reasons for DFID’s focus on low-income countries, but do you see more of a reallocation in funding towards middle-income countries in the future?
NY: The reason we focus on low-income countries primarily is because this is enshrined in law in the UK. DFID is required by the International Development Act to focus on the poorest countries and we have, in our Bilateral Aid Review, shifted our aid allocations to focus on those where we can make the most difference to the poorest and most vulnerable people – including rapid growth in ‘fragile states’. DFID’s largest bilateral programmes are in countries such as Ethiopia, Nigeria and Pakistan. Sometimes, however, the best ways to tackle poverty include promoting growth and trade relationships and tackling conflict and humanitarian issues and for this reason and others we also recognize the importance of working with emerging economies and a wide range of partners - our recent work in Libya is an example of our presence in the Middle East and North Africa.
DI: Another issue we face as researchers is misalignment between funding (and reporting) cycles and impact evaluation cycles. When evaluations are built around government policies, the timeline becomes tied to government implementation timelines which often experience multiple delays, which when added to the time needed for interventions to show effects, could mean 4-5 years is not that uncommon for an evaluation timeframe. What is your approach to funding evaluations to reduce the amount of mismatch between funding cycles and evaluation timelines?
NY: This is a really important point. We would often seek to have a separate programme budget and business case/procurement process for the evaluation, which –apart from ensuring some separation and avoiding conflict of interests - allows work to start in advance of the programme itself (as required for researching evaluation questions, developing the evaluation design and baseline survey) and after the programme has finished (to measure sustainable impact over the long term).
DI: One view is that funders should think of themselves like venture capitalists – investing in a portfolio of evaluations knowing that many are risky and likely to fail – but that others will succeed and have high payoffs. However, we have found some resistance to this view, and a natural conservatism among some donors towards funding projects which are further along and have lower risks (but also therefore never funding some of the really innovative untested and unproven ideas). What is DFID’s view on this trade-off?
NY:There is quite a lot of sympathy in DFID for this view - having a reasonable appetite towards risk, provided the risks are identified and managed, and promoting innovation are all high up our agenda. The fact that we are willing to invest heavily in post-conflict environments, where the risks are significant but the returns are also potentially highest in terms of transforming lives, speaks for itself. A good example of this was the way the UK went into Rwanda early and provided long-term support after the genocide, as well as our programmes in Mozambique. Both have paid off substantially. We currently have an important programme in Burma.
DI: How are you evaluating your funding of evaluations?
NY: A good question! This is an area which needs quite a lot more attention in DFID, learning from our experience with evaluating our research programmes. We are currently involved in the early stages of an evaluation of 3ie. The point you made earlier about taking a portfolio approach to this is important – the impacts of research and evaluation are not always where you expect them to be, and some will pay off with game-changing effects while others take years to make a difference or the difference they make is indirect.
Join the Conversation