Estimating the direct and indirect benefits of transport projects remains difficult. Only a handful of rigorous impact evaluations have been done as the methodologies are technically and financially demanding. There are also differences between the impact of rural and urban projects that need to be carefully anticipated and evaluated.
Can we simplify the methodologies?
Despite the Bank’s rich experience with transport development projects, it remains quite difficult to fully capture the direct and indirect effects of improved transport connectivity and mobility on poverty outcomes. There are many statistical problems that come with impact evaluation. Chief among them, surveys must be carefully designed to avoid some of the pitfalls that usually hinder the evaluation of transport projects (sample bias, timeline, direct vs. indirect effects, issues with control group selection, etc.).
Impact evaluation typically requires comparing groups that have similar characteristics but one is located in the area of a project (treatment group), therefore it is likely to be affected by the project implementation, while the other group is not (control group). Ideally, both groups must be randomly selected and sufficiently large to minimize sample bias. In the majority of road transport projects, the reality is that it is difficult to identify control groups to properly evaluate the direct and indirect impact of road transport improvements. Also, road projects take a long time to be implemented and it is difficult to monitor the effects for the duration of a project on both control and treatment groups. Statistical and econometric tools can be used to compensate for methodological shortcomings but they still require the use of significant resources and knowhow to be done in a systematic and successful manner.
Final (for now) evaluationtastic installment on Oxfam’s attempts to do public warts-and-all evaluations of randomly selected projects. This commentary comes from Dr Jyotsna Puri, Deputy Executive Director and Head of Evaluation of the International Initiative for Impact Evaluation (3ie)
Oxfam’s emphasis on quality evaluations is a step in the right direction. Implementing agencies rarely make an impassioned plea for evidence and rigor in their evidence collection, and worse, they hardly ever publish negative evaluations. The internal wrangling and pressure to not publish these must have been so high:
- ‘What will our donors say? How will we justify poor results to our funders and contributors?’
- ‘It’s suicidal. Our competitors will flaunt these results and donors will flee.’
- ‘Why must we put these online and why ‘traffic light’ them? Why not just publish the reports, let people wade through them and take away their own messages?’
- ‘Our field managers will get upset, angry and discouraged when they read these.’
- ‘These field managers on the ground are our colleagues. We can’t criticize them publicly… where’s the team spirit?’
- ‘There are so many nuances on the ground. Detractors will mis-use these scores and ignore these ground realities.’
The zeitgeist may indeed be transparency, but few organizations are actually doing it.
As part of a new series looking how institutions are approaching impact evaluation, DI virtually sat down with Nick York, Head of Evaluation and Gail Marzetti, Deputy Head, Research and Evidence Division
In a New York Times column last Friday David Brooks discussed a book by Jim Manzi, and extolled the idea of randomized field trials as a way for the US to make better policies.
While it’s nice to welcome Citizen Brooks into the fold, there are a couple of points in his article worth exploring a bit.
Building partly on a previous post on the value of indices, I'm highlighting this week a new edited volume published by Peter Lang Press, entitled Measures of Press Freedom and Media Contributions to Development: Evaluating the Evaluators. This rich and informative collection of essays, edited by Monroe Price, Susan Abbott and Libby Morgan, focuses a spotlight on well known indices in the area of press freedom and media independence, raising valuable questions about what the indices are measuring, what they are not measuring, and the linkage between assistance to independent media and democratization. I've contributed a chapter to this volume, as have expert colleagues such as Guobin Yang, Andrew Puddephat, Lee Becker and Tudor Vlad, Craig LaMay, fellow CommGAP blogger Silvio Waisbord, and many others.
Courts must expeditiously, but fairly, adjudicate corruption cases, and the penalties imposed on those convicted must be sufficient to dissuade others from similar acts. To ensure that anti-corruption laws are indeed being effectively enforced, governments need to monitor the enforcement process.
Doing so can provide performance measures to inform and guide policy design and implementation. These performance measures also serve as indicators of corruption. In the short run, policy makers may not be able to do much to change these indicators, but measures, focused on performance, can provide a country something more concrete to act upon, helping policy-makers to prioritize.
For example, if the number of completed corruption investigations in a particular country is low because of difficulty in obtaining evidence, it can identify changes in policy and procedures which expand or strengthen investigators powers and tools such as providing it with subpeona powers or access to financial records.
The newly launched IEG Annual Review of Development Effectiveness 2009 attests the World Bank a significant increase in development effectiveness from financial year 2007 to 2008. After a somewhat disappointing result last year, 81 % of the development projects that closed in fiscal 2008 were rated satisfactory with regard to the extent to which the operation's major relevant objectives were achieved efficiently.
One crux remains: the measurement of impact. Monitoring and evaluation components in development projects are by far not as frequent as IEG would wish: Two thirds of the projects in 2008 had marginal or negligible M&E components. Isabel Guerrero, World Bank Vice President of the South Asia Region, listed several reasons at the launch of the IEG report this week: the lack of integrative indicators, the Bank's tradition to measure outputs instead of outcomes, the lack of baseline assessments in most projects, and reluctance on the clients' side to realize M&E in projects.
While their booths were temporarily closed during the exhibition, finalists, along with development community representatives, participated in Knowledge Exchange technical assistance workshops -- like this one on capacity building.
Three DM2008 jurors who are past grant winners are sharing their well-learned lessons with the hundred finalists.
Take 2006 winner Florence Cassassuce (in photo at right), who brought her water-purifying UV-light bucket to 900 villagers on the rural outskirts of La Paz in Baja California, Mexcio. Cassassuce, implementing her project with the advice of World Bank Senior Environmental Specialist Ricardo Hernandez Murillo, installed 3,500 buckets toward the goal of 6,00, ahead of schedule. But the original buckets didn't always work well, especially in the field, and improvements had to be made with better, and faster, plastic-injection manufacturing.
As the recently named Director of the Agriculture and Rural Development Department -- one of DM2008's funders --Juergen Voegele is leading a vigorous effort to re-energize and broaden the World Bank's commitment to agricultural development. During his peripatetic rounds of the competition, Voegele sat down for this mini-interview:
Q. What are your impressions of what you see on the floor of the exhibition?