I was recently invited to speak at the biannual infrastructure retreat of the IADB and was excited to learn that they had decided to devote two days of their retreat to discussing the development of an impact evaluation (IE) program in the transport sector. This is largest sector in most development banks, yet one that has not caught the IE bug. Perhaps this is because there is a perception that IE is difficult or impossible to incorporate into transportation projects. I thus set out to survey some of the literature in this sector, understand the state of knowledge and find out what gaps we could work on if a program were to take off. What I found is that far from being impossible, creative approaches to evaluating transport investments can teach us quite a lot about their value and untapped potential.
There are a number of recent studies by Donaldson and coauthors that emphasize the large payoffs from developing large transport networks. He looks at the construction of India’s railroad network and its important effect on reducing trade costs and inter-regional price gaps, and increasing trade and raised real income (Donaldson, 2012). Donaldson’s collaboration with Hornbeck (2013) takes us back in time to the expansion of the US railroad network from 1870 to 1890, which improved market access and county agricultural land values. Lest I forget, Banerjee, Duflo and Qian (2012) demonstrated an income effect in China associated with proximity to a (potential) transport corridor. A more recent study by Yu Qin (2014) warns us that innovations in transportation may bring important distributional consequences as well. She estimates that a high-speed rail upgrade in China may have reduced GDP and GDP per capita, by 4-6 percent, in counties that are now bypassed by the trains.
Nonetheless, gains from transportation are clearly important as perhaps are gaps in investment in developing countries. A recent study utilizing CPI micro-data (Atkin and Donaldson, 2014) exposed that the cost of trading over a unit distance in Ethiopia or Nigeria is four to five times larger than within the US. No wonder our data collection in Africa is so expensive! This stylized fact should give us pause for thought. Returns to transport infrastructure should be high. However the evidence is hard to come by. In a much quoted study, Ren and De Walle (2011) estimate positive impacts of rural road rehabilitation on market development at the commune level in rural Vietnam. In one of the few RCTs in transport, Gonzalez-Navarro et al. (2010) show that street paving in Mexico raises housing values by 16 percent and land values by 54 percent. In Medellín, Colombia, the public transit system to connect isolated low-income neighborhoods to the city's urban center, lowered violence (Cerdá et al., 2012). The decline in the homicide rate is 66 percent greater in intervention than in the control neighborhoods. However notable these studies are, they in no way measure up to the amount of investments we are making in this sector. To start closing the gap, we are conducting two impact evaluations of World Bank projects in Nigeria and Rwanda. We’re collecting comprehensive surveys to understand the range of outcomes road projects can affect. But this is still just the tip of the iceberg in terms of fully understanding the impacts from the sector.
And then there are the driverless cars...In addition to the payoffs, we are interested in understanding how to increase returns on investment and what contributions innovation and leapfrog technology can make. For example, can driverless car technology reduce fatalities and injuries and save travel time (Winston and Mannering, 2014)? What are the returns to instituting bicycle lanes when one percent of population uses it? What about when that demand increases to 20 percent? And speaking of the leapfrog technology, a recent study found that here in Washington, DC, subsidized access to the public transit system increases the probability of finding employment (within 40 days) by nine percentage points (Phillips, 2014). Let’s also not forget the regulatory aspect in all of this. Consider the study (Salas, 2010), that noted how driving restrictions in Mexico reduced air pollution 12 to 18 percent and how drunk-driving laws in Brazil (Lei Seca No.11.705, Brazil) reduced mortality by 7.4 percent (Malta et al. 2010). Whether we are interested in public health or the environment, figuring out how to get people to buckle up or use the bus is critical.
There’s also another facet to consider in our line of work – the returns (and sustainability) to our investments depend on their cost, quality and durability. For instance, in his seminal study in Indonesia, Olken, (2008) finds that top-down audits reduced “missing expenditures” in road construction by eight percentage points, while bottom-up participatory monitoring did not. More importantly, thousands of contracts go out every year across the globe to build, rehabilitate and maintain roads and we know almost nothing about how to improve procurement and contractual arrangements to decrease cost and increase quality. De Silva et al., (2008) estimate that revealing engineering cost estimates to prospective bidders reduces average bid price, especially when there is cost uncertainty. In Japan, Ohashi (2006) estimates a three percent reduction in procurement cost from pre-selecting suppliers. The latter two studies clearly emphasize that procurement rules matter in securing better tenders. Yet, these few results are a reminder as to how little evidence we have on what type of contracting, property rights distribution, governance arrangements and payment systems would ensure the sustainability of our investments.
Closing the evidence gap and factoring in returns on investment are two of the many outcomes from use of IE tools. IE can help zero in on demand and supply factors that determine success and help increase returns and sustainability of investments. IE helps us to understand the weak or missing links and support experiments to find the right solutions and inform policy. I ask you to weigh in. What further experiments should we be considering? What are the missing problem statements as we move ahead to incorporate more IE into transportation? Our own World Bank Global Practice on Transport is moving full steam ahead with its impact evaluation program and is interested in knowing the answers.
Addendum: Response from Arianna to comments on this post
There are a number of recent studies by Donaldson and coauthors that emphasize the large payoffs from developing large transport networks. He looks at the construction of India’s railroad network and its important effect on reducing trade costs and inter-regional price gaps, and increasing trade and raised real income (Donaldson, 2012). Donaldson’s collaboration with Hornbeck (2013) takes us back in time to the expansion of the US railroad network from 1870 to 1890, which improved market access and county agricultural land values. Lest I forget, Banerjee, Duflo and Qian (2012) demonstrated an income effect in China associated with proximity to a (potential) transport corridor. A more recent study by Yu Qin (2014) warns us that innovations in transportation may bring important distributional consequences as well. She estimates that a high-speed rail upgrade in China may have reduced GDP and GDP per capita, by 4-6 percent, in counties that are now bypassed by the trains.
Nonetheless, gains from transportation are clearly important as perhaps are gaps in investment in developing countries. A recent study utilizing CPI micro-data (Atkin and Donaldson, 2014) exposed that the cost of trading over a unit distance in Ethiopia or Nigeria is four to five times larger than within the US. No wonder our data collection in Africa is so expensive! This stylized fact should give us pause for thought. Returns to transport infrastructure should be high. However the evidence is hard to come by. In a much quoted study, Ren and De Walle (2011) estimate positive impacts of rural road rehabilitation on market development at the commune level in rural Vietnam. In one of the few RCTs in transport, Gonzalez-Navarro et al. (2010) show that street paving in Mexico raises housing values by 16 percent and land values by 54 percent. In Medellín, Colombia, the public transit system to connect isolated low-income neighborhoods to the city's urban center, lowered violence (Cerdá et al., 2012). The decline in the homicide rate is 66 percent greater in intervention than in the control neighborhoods. However notable these studies are, they in no way measure up to the amount of investments we are making in this sector. To start closing the gap, we are conducting two impact evaluations of World Bank projects in Nigeria and Rwanda. We’re collecting comprehensive surveys to understand the range of outcomes road projects can affect. But this is still just the tip of the iceberg in terms of fully understanding the impacts from the sector.
And then there are the driverless cars...In addition to the payoffs, we are interested in understanding how to increase returns on investment and what contributions innovation and leapfrog technology can make. For example, can driverless car technology reduce fatalities and injuries and save travel time (Winston and Mannering, 2014)? What are the returns to instituting bicycle lanes when one percent of population uses it? What about when that demand increases to 20 percent? And speaking of the leapfrog technology, a recent study found that here in Washington, DC, subsidized access to the public transit system increases the probability of finding employment (within 40 days) by nine percentage points (Phillips, 2014). Let’s also not forget the regulatory aspect in all of this. Consider the study (Salas, 2010), that noted how driving restrictions in Mexico reduced air pollution 12 to 18 percent and how drunk-driving laws in Brazil (Lei Seca No.11.705, Brazil) reduced mortality by 7.4 percent (Malta et al. 2010). Whether we are interested in public health or the environment, figuring out how to get people to buckle up or use the bus is critical.
There’s also another facet to consider in our line of work – the returns (and sustainability) to our investments depend on their cost, quality and durability. For instance, in his seminal study in Indonesia, Olken, (2008) finds that top-down audits reduced “missing expenditures” in road construction by eight percentage points, while bottom-up participatory monitoring did not. More importantly, thousands of contracts go out every year across the globe to build, rehabilitate and maintain roads and we know almost nothing about how to improve procurement and contractual arrangements to decrease cost and increase quality. De Silva et al., (2008) estimate that revealing engineering cost estimates to prospective bidders reduces average bid price, especially when there is cost uncertainty. In Japan, Ohashi (2006) estimates a three percent reduction in procurement cost from pre-selecting suppliers. The latter two studies clearly emphasize that procurement rules matter in securing better tenders. Yet, these few results are a reminder as to how little evidence we have on what type of contracting, property rights distribution, governance arrangements and payment systems would ensure the sustainability of our investments.
Closing the evidence gap and factoring in returns on investment are two of the many outcomes from use of IE tools. IE can help zero in on demand and supply factors that determine success and help increase returns and sustainability of investments. IE helps us to understand the weak or missing links and support experiments to find the right solutions and inform policy. I ask you to weigh in. What further experiments should we be considering? What are the missing problem statements as we move ahead to incorporate more IE into transportation? Our own World Bank Global Practice on Transport is moving full steam ahead with its impact evaluation program and is interested in knowing the answers.
Addendum: Response from Arianna to comments on this post
Thank you all for the comments and emails that you’ve sent on this post. Of note was an email I received from my colleague, Harris Selod. He writes of the need to measure the cost associated with non-physical factors of transport including those tied to market structure, borders and regulation and the importance of experimenting with different types of mechanisms to do so. The World Bank is starting to measure those costs with several studies in Africa and LAC (Teravaninthorn and Raballand 2008, Lall et al. 2009, World Bank 2012, World Bank 2014).
My mention of the DC metro subsidy triggered discourse on the impact of transport on labor market outcomes. Harris just published a piece in the Handbook of Regional Science that discusses the disconnect between jobs and residential locations (also known as “spatial mismatch”) and how it aggravates unemployment. This is another area where experimentation (and specifically RCTs) could help us sort through the channels at work, (considering such questions as commuting cost, job search cost, job search efficiency, decreased productivity etc.), and the relative magnitude of the effects, to guide policy prioritization.
Much has yet to be done on comparing the efficiency of different instruments to reduce congestion and curb transport-induced pollution, from investments in public transport infrastructure, to pricing, and regulation. Timilsina has looked at the role of pricing (Anas and Timilsina 2009, Parry andTimilsina 2010, Timilsina and Dulal, 2011).
And finally Harris also refers us to another study on the distributional impacts of transport investments in China by Mark Roberts and Uwe Deichmann et al. (2012) who estimated a structural geography model to demonstrate that highway investment benefited some prefectures at the expense of others. In the future, developing spatial “general equilibrium” approaches could help us assess the more systemic impact of transport investments beyond the focus on the local impact of projects.
Join the Conversation