Syndicate content

Impact evaluation

The need to improve transport impact evaluations to better target the Bottom 40%

Julie Babinard's picture
In line with the World Bank’s overarching new goals to decrease extreme poverty to 3 % of the world's population by 2030 and to raise the income of the bottom 40% in every country, what can the transport sector do to provide development opportunities such as access to employment and services to the poorest?

Estimating the direct and indirect benefits of transport projects remains difficult. Only a handful of rigorous impact evaluations have been done as the methodologies are technically and financially demanding. There are also differences between the impact of rural and urban projects that need to be carefully anticipated and evaluated.

Can we simplify the methodologies?

Despite the Bank’s rich experience with transport development projects, it remains quite difficult to fully capture the direct and indirect effects of improved transport connectivity and mobility on poverty outcomes. There are many statistical problems that come with impact evaluation. Chief among them, surveys must be carefully designed to avoid some of the pitfalls that usually hinder the evaluation of transport projects (sample bias, timeline, direct vs. indirect effects, issues with control group selection, etc.).

Impact evaluation typically requires comparing groups that have similar characteristics but one is located in the area of a project (treatment group), therefore it is likely to be affected by the project implementation, while the other group is not (control group). Ideally, both groups must be randomly selected and sufficiently large to minimize sample bias. In the majority of road transport projects, the reality is that it is difficult to identify control groups to properly evaluate the direct and indirect impact of road transport improvements. Also, road projects take a long time to be implemented and it is difficult to monitor the effects for the duration of a project on both control and treatment groups. Statistical and econometric tools can be used to compensate for methodological shortcomings but they still require the use of significant resources and knowhow to be done in a systematic and successful manner.

Africa Impact Evaluation Podcast: Economic Empowerment of Young Women in Africa #AfricaBigIdeas


When it comes to helping young women in Africa with both economic and social opportunity, what does the evidence tell us?  Broadcaster Georges Collinet sat down with researchers and policymakers to discuss the hard evidence behind two programs that have succeeded in giving girls a better chance at getting started in their adult lives.

Are Impact Evaluations Enough? The Social Observatory Approach to Doing-by-Learning

Vijayendra Rao's picture

Impact Evaluations are just one of many important tools to improve “adaptive capacity.” To improve implementation, they need to be integrated with monitoring and decision support systems, methods to understand mechanisms of change, and efforts to build feedback loops that pay attention both to everyday and long-term learning.  While there has been some scholarly writing and advocacy on this point, it has been more talk than action. 

Hammers, wrenches, and development policy

Markus Goldstein's picture
The old saw goes: when you have a hammer, everything looks like a nail.    But what if the best way to fix your broken policy is actually a bolt?   I was recently at a workshop where someone was presenting preliminary results of an evaluation cash transfer program which, while perhaps started with social protection kind of objectives in mind, actually seems to have had impacts on business creation and revenues that dwarfed your average business training program or microfinance program. 
 

The International Rescue Committee's approach to impact evaluation

Markus Goldstein's picture
Our series on institutional approaches to impact evaluation continues!    DI recently virtually sat down with Jeannie Annan, Director of Research and Evaluation at the International Rescue Committee.   
 
DI:  What is the overall approach to impact evaluation at the IRC?
JA:  We are committed to providing (or supporting) the most effective and cost-effective interventions. This means using the best available research about what works combined with understanding of the context and experience in implementation to design and deliver our programs.  

The MCC's approach to impact Evaluation

Markus Goldstein's picture
In our continuing series on discussing institutional approaches to impact evaluation, DI virtually sat down with Jack Molyneaux, Director of Independent Evaluations at the Millennium Challenge Corporation.  (Please note that these are Jack’s opinions, not that of the MCC)
 
DI:  Impact Evaluation seems to be something that's pretty important at the MCC. Can you tell us a bit about how this focus came about? 
JM: Since its inception MCC’s mandate has included demonstrating results.  Rigorous impact evaluations have always been a key component of that mandate. 

Weekly links July 12: daycare, remittances for education, opaque measurement, funding, and more…

David McKenzie's picture
  • How should we measure what is a high-income country? Martin Ravallion explains and critiques the World Bank definition on the CGD blog.
  • Aid Thoughts discusses new work on the value of daycare in Brazilian slums.
  • A new From Evidence to Policy note looks at the long-term impact of a conditional cash transfer on education in Colombia-part of the analysis uses admin data on test scores for graduating students – “students whose families received cash grants were between 4 and 8.4 percentage points more likely to graduate from high school; but Students whose families received the cash grants didn’t score higher on the national standardized achievement test given a year before graduation”.
  • Classic papers in behavioral finance summed up in a few sentences – Noah Smith gives his take on essential papers in behavioral finance.
  • On the IDB Development Effectiveness blog, Dean Yang and co-authors summarize their new study on the use of matching funds to channel remittances towards education in El Salvador.
  • Funding opportunity: The World Bank’s Strategic Impact Evaluation Fund (SIEF) has a new call for proposals for work on basic education, water and sanitation, early childhood development, and health systems. Details here.
  • Funding opportunity: 3ie has funding available under an agricultural innovation thematic window. This grant window will fund up to 16 new impact evaluations of interventions in the areas of knowledge transfer, contractual arrangements, adoption, and soil health
  • Funding opportunity: (Not just for impact evaluations) IZA and DFID are now accepting applications for funding in Phase III of the Growth and Labor Markets in Low Income Countries (GLM | LIC) program.  This will fund work on 1. Growth and labor market outcomes, 2. Active labor market policies, 3. Labor market institutions, 4. Migration and labor markets, 5. Gender and 6. Data for labor market analysis. Application materials here.

Public randomization ceremonies

Damien de Walque's picture

With Jake Robyn* and Gaston Sorgho**

Randomization might- at first – sound like a scary word for health policy makers and professionals. They read medical journals and know from their training that randomized trials are scientifically rigorous designs to evaluate the impact of a program. But their first inclination might be to prefer to have the randomized trial in somebody else’s backyard. Randomization seems politically difficult. How to explain it to the people who will have to wait for the new intervention? Will it not create a backlash with the people who are randomly assigned to the control group? How will the population be convinced that the random allocation was fair and that there were no back room deals?

Our experience in many countries is that public randomization ceremonies are an excellent platform to build support for randomization and for the entire impact evaluation process. In Cameroon, we organized public randomization ceremonies in three Regions to assign health facilities to four study groups in an impact evaluation of performance-based financing (PBF) in the health sector. Held in the regional capitals and combined with the official launching of the project in each Region, we invited representatives of each facility, district health management teams, and local government, who all took part themselves in the randomization. Each of the randomization ceremonies received close oversight from the central and regional levels of the Ministry of Public Health. This made the randomization process completely fair and transparent to all health facilities participating in the study.

Introducing the Africa Gender Innovation Lab

Markus Goldstein's picture

Today I wanted to take the opportunity to talk about a new initiative that the Africa Region and the Research Group at the World Bank are launching today.   The idea here is that we don't know enough about how to effectively address the underlying causes of gender inequality. Let me start by explaining what I mean by underlying causes.    Take the case of female farmers.    There is a lot of literature out there which shows that women have lower agricultural yields than men.   And some of it shows that this is because women have lo


Pages