Syndicate content

monitoring and evaluation

Beyond ribbon-cutting: measuring the real impact of transport projects

Nancy Vandycke's picture
Photo: World Bank/Flickr
Development practitioners often rely on Monitoring and Evaluation (M&E) performance indicators to assess the results of a transport project. Collecting indicators before, during, and after a project allows us to gain insights about project execution and project outputs, which can help us, for example, measure changes in travel time or Bus Rapid Transit (BRT) system ridership. While this approach is important, well anchored into project design, and quite practical, it is not intended to evaluate “impact”. Observed changes in outcomes cannot be attributed to the project: many other external factors, such as economic conditions, interrelated policies or projects, or seasonal trends, also come into play. In other words, a descriptive approach fails to establish causality between a project or intervention and subsequent outcomes such as changes in income, labor markets, quality of life, or market efficiency.

To overcome the limitations of traditional M&E, the development community is increasingly turning to impact evaluation, an alternative approach whose methods more directly address the issue of causality. In that context, the World Bank’s transport experts have partnered with colleagues from the Development Impact Evaluation (DIME) team to rethink the way the impact of transport is measured. Two years ago, with support from the UK Department for International Development (DFID), a transport-dedicated impact evaluation program was launched: “IE Connect for Impact”. Now, impact evaluation is being implemented on 10 projects, covering rural roads, urban mobility, transport corridor development, and road safety. More projects will be selected toward the end of the year, as part of Phase II of the program.

The expected benefits are clear: informing project delivery during design and implementation, documenting the effects of policy and investment interventions, and prioritizing and filling knowledge gaps in the sector. Despite these significant benefits, transport accounts for less than 1% of all impact evaluation work —a very low proportion compared to the weight of other sectors such as in health (65% of all published impact evaluations), education (23%), agriculture and rural development (10%), or water (4%).

The need to improve transport impact evaluations to better target the Bottom 40%

Julie Babinard's picture
In line with the World Bank’s overarching new goals to decrease extreme poverty to 3 % of the world's population by 2030 and to raise the income of the bottom 40% in every country, what can the transport sector do to provide development opportunities such as access to employment and services to the poorest?

Estimating the direct and indirect benefits of transport projects remains difficult. Only a handful of rigorous impact evaluations have been done as the methodologies are technically and financially demanding. There are also differences between the impact of rural and urban projects that need to be carefully anticipated and evaluated.

Can we simplify the methodologies?

Despite the Bank’s rich experience with transport development projects, it remains quite difficult to fully capture the direct and indirect effects of improved transport connectivity and mobility on poverty outcomes. There are many statistical problems that come with impact evaluation. Chief among them, surveys must be carefully designed to avoid some of the pitfalls that usually hinder the evaluation of transport projects (sample bias, timeline, direct vs. indirect effects, issues with control group selection, etc.).

Impact evaluation typically requires comparing groups that have similar characteristics but one is located in the area of a project (treatment group), therefore it is likely to be affected by the project implementation, while the other group is not (control group). Ideally, both groups must be randomly selected and sufficiently large to minimize sample bias. In the majority of road transport projects, the reality is that it is difficult to identify control groups to properly evaluate the direct and indirect impact of road transport improvements. Also, road projects take a long time to be implemented and it is difficult to monitor the effects for the duration of a project on both control and treatment groups. Statistical and econometric tools can be used to compensate for methodological shortcomings but they still require the use of significant resources and knowhow to be done in a systematic and successful manner.