Syndicate content

Is “impact evaluation” the right term?

Jed Friedman's picture

Today’s midweek post on the Development Impact blog is a bit lighter fare than the previous two meaty posts on external validity issues from David and Berk. It concerns the very term that is the focus of this blog: impact evaluation.

Yesterday Berk, I, and other research colleagues attended a day long discussion with operational colleagues in the Africa region on the role of impact evaluation activities under World Bank projects. Many challenges of how to “mainstream” impact evaluation into lending projects were discussed and debated. These challenges include the identification of funding sources, the proper forums for dissemination, and, equally important, the confusion in the wider development community over what is mean by “impact evaluation”.

One colleague relayed “every time I hear the term impact evaluation I think of ‘car accident’“. On reflection, impact evaluation is certainly not an elegant term. To a layperson the term may connote the activity that an insurance company must undertake in the aftermath of a hurricane or other widespread disaster. From the perspective of certain other disciplines, it strikes practitioners as ill-defined and vague.

At a discussion table, several health colleagues conveyed various definitions of impact evaluation. These included the view that impact evaluation focuses solely on ultimate outcomes such as, in the case of health, mortality (and studies that focus on process indicators such as drug availability should not be considered impact evaluation). Apparently many counterparts in government share this view, as well as the view that impact evaluation is necessarily an expensive and lengthy process because it exclusively focuses on long-term impacts. Another common definition is that impact evaluation studies are necessarily randomized trials.

These varied understandings often lead to real initial problems when working across disciplines and when introducing the possibility of impact evaluation into project design discussions with government counterparts. Sometimes an initial misunderstand will color a whole series of discussions and may ultimately affect the decision of whether to go forward with a study.

For the community of researchers in which I engage, impact evaluation is a catch-all term for any research that seeks to attribute the causal effect that a program, policy, or intervention has on outcomes or indicators of interest.  It is the emphasis on rigorous causal attribution in an empirical setting that separates the activities that collectively fall under impact evaluation from other research or monitoring activities.

While clearly not universal, this definition is fairly standard among applied social science researchers: it’s echoed in this recent book and by the 3ie for example.

The choice of outcome of interest is driven by the question at hand and can involve ultimate long-term outcomes or simple short-term process indicators. The method of evaluation can be randomization, but can also exploit discontinuities, etc. The time frame can span many years or a matter of days. The study setting can be highly controlled as in an efficacy trial or fully immersed in the messy world of at-scale programs, as in effectiveness trials. The particular question under study, and the setting in which it takes place, will determine all of these factors.

My colleagues at the table suggested other applicable terms that may better describe the same activities under discussion: operational research, evaluative research, and implementation science. Here’s my take on each:

Operational research – I see the appeal of this term because the IE work under bank projects often involves understanding the impact of alternative modes of program implementation – a very important type of program effectiveness research. Operational research with rigorous causal attribution is a key subset of impact evaluation activities. However I don’t believe this term encapsulates the full scope of IE activities, such as research under highly controlled settings that seeks to elucidate important aspects of human behavior. One example of this type of research is a recent paper by Cohen and Dupas that estimates a demand curve for malaria bed nets.

Evaluative research – This isn’t a bad term. It is sufficiently vague and still conveys the activity of evaluation. But I suppose it missed the bus in that “impact evaluation” has much more widespread usage today.

Implementation science – Similar to operational research, it has a hands-on appeal for many of the effectiveness-types studies we do. However as argued above, impact evaluation is a wide tent that encompasses studies that focus on program implementation, but other study topics as well.

After this table discussion, we took a poll of the preferred term to describe the research activities that were under discussion throughout the day. The final tally:

Operational research: 2 votes

Evaluative research:  2 votes

Implementation science: 3 votes

Impact evaluation: 2 votes

The jury did not reach a verdict, deliberations must continue.

Comments

Submitted by Anonymous on
Thanks for rasing the question loud an clear! The current World Bank use of the term "impact evaluation" has, in my experience, indeed lead to repeated misunderstandings in conversations on results monitoring etc. with partners and couterparts - and triggered the need for me explaining different concepts of the term "impact", and the particular approach of the World Bank on "impact evaluation" which is different from the way most other development cooperation stakeholders see it. The OECD Development Co-operation Directorate had issued, in 2002, a glossary of key terms in evaluation and results-based management, which, I believe, present standard definitions which are broadly recognized and accepted in international development cooperation - except, as it seems, (in parts) by the World Bank. http://www.oecd.org/dataoecd/29/21/2754804.pdf Impacts are defined as "Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended." The glossary might be useful as a source of possible options for the long overdue re-definition of the work done by World Bank staff under the heading "impact evaluation".

It is always difficult to harmonize terms across different disciplines - I am reminded of analytic methods that are virtually the same in biostatistics and economics but are called by completely different names. Confusion typically ensues when practitioners in these different disciplines meet. The OECD glossary is a very good resource to hopefully help guide harmonization of terminology, but I expect this will be a slow process with further misunderstandings down the road...

Jed This is indeed a can of worms.. I work as an adviser to programmes around the Horn, East and Central Africa and my area of work is primarily around (gulp) Impact! I too tend to go with the definition that 'anonymous' gave above.. However, with a huge proportion of programming in conflict (Somalia, DRC) and high vulnerability (Ethiopia, Northern Kenya, South Sudan) areas the question that is often asked is "How realistic are our expectations of long term impact?" With the control that NGOs have rapidly diminishing as they move up the input-output-outcome-impact chain, expecting any impact becomes a slightly pointless exercise.. I have now started dropping 'long-term' from the very definition of impact and often advise teams in humanitarian programming to focus on shorter term (perhaps not as long lasting) impacts which would actually be better off being called outcomes.. This may sound sophistry but....

... I agree that the term "Impact" has had a long-standing definition in results-chain frameworks that now "Impact Evaluation" is turing on its head. Or rather dramatically expanding in meaning to potentially encompass outcomes and outputs, depending on the setting. I imagine in the long-run these confusions will sort themselves out, but you know what they say about the long-run... Jed.

Submitted by Isabela on
As an engineer, who does not associate impact evaluation only with extreme events, I would vote for keeping the current name. Operational research and Implementation science seem to dillute the scope of IE and further complicate its understanding. Evaluative research has a lab feeling associated to it, which would contradict its operational value.

Re the OECD DAC definition of impact: "Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended." Interpreted literally this focus on the long term could mean that any changes during a project implementation period, commonly three years, would not be considered to be a kind of impact. This could be a bit of a problem in projects aimed at reducing infant or child mortality by improving the quality of health service provision, for example. In the case of maternal and infent health projects it would be commonly expected that some reductions in the numbers of cases of child and maternal deaths would occur within a project's lifespan. In fact the sooner these kinds of changes happened the better. However, if impact was defined in a different way then kind of this perverse anomaly would not arise. Impact could be defined in terms of social rather than temporal distance, as changes in in the lives of people of ultimate concern. In this case it is mothers and infants, who are at the end of one or more chains of actors, through which aid resources and their effects flow. The Social Framework (see link below) uses this concept of social distance, whereas the vertical structure of the Logical Framework uses either temporal distance, or a conflation of social and temporal distance. http://mande.co.uk/2008/uncategorized/the-social-framework-as-an-alternative-to-the-logical-framework-2/