One of the interesting discussions I had this last week was with a World Bank consultant trying to think about how to evaluate the impact of large-scale infrastructure projects. Forming a counterfactual is very difficult in many of these cases, and so the question is what one could think of doing. Since I get asked similar types of questions reasonably regularly, I thought I’d share my thoughts on this issue, and see whether anyone has good examples to share.
Certain types of infrastructure projects may lend themselves to more traditional impact evaluation methods. This is particularly the case when infrastructure is rolled out over time and geographic space, allowing difference-in-difference or IV methods to compare places with and without service (good examples include Taryn Dinkelman’s work on electrification in South Africa, Esther Duflo and Rohini Pande’s work on dams in India, and Ren Mu and Dominique van de Walle’s work on roads in Vietnam). These approaches are easier in larger countries, where there might be many dams or miles of roads being constructed, and then the evaluation challenge is similar to that in much of what we do: coming up with good instruments or convincing the reader that the matched difference-in-differences approach is giving a reasonable counterfactual.
The much more challenging case to think about is where the project is something like developing a single new port, or a new airport, or one large new powerplant in a small country. What can we do in such cases?
Approach One-comparing more and less affected units: Suppose we want to see whether opening a new port has increased exports. One approach would be to identify firms in industries which usually depend on sea transport for exports (based on countries with well functioning ports and other infrastructure) and firms in industries which typically export through air freight or other means. Then a Rajan-Zingales type approach can be used, to see whether exports grow more rapidly in these port-dependent industries after the opening of the port than in the other industries. The counterfactual in this case is thus export growth in other industries in the country – this takes care of macroeconomic factors in the country as a whole, but not industry specific shocks taking place.
Approach Two – constructing a counterfactual based on similar countries and time series data:
An alternative way of defining a counterfactual might be to assume that in the absence of the port, exports in a particular sector would grow in a similar way to export growth in similar countries who also have limited port access. Or in a large country, to firms in other states within the country that are close to ports. This takes care of industry-specific shocks taking place (e.g. cocoa exports might be booming globally), but ignores country-specific shocks.
Combining approaches one and two then offers the possibility of controlling for both industry-specific and country-specific shocks, and providing a somewhat credible counterfactual. The main concerns are then going to be whether one can put together consistent data across countries and industries, and whether there are enough sector/country units to provide sufficient degrees of freedom. Increasing the time dimension may improve power further here (see my more T paper).
Approach Three – think like an economic historian, and work through the causal chain.
Some outcomes might be harder to measure using approaches one and two, or the necessary data for doing so may not be available. A complimentary approach is to think through the chain of logic needed for the intervention to have the effect of interest, and to set up falsifiable predictions along the way. For example, how might ports affect exports? We might think that in order for a sizeable effect to take place, we need all of the following to occur: i) the cost of transporting cargo must fall in the country, and perhaps we can measure how much by; ii) trade as a whole grows, not just substitution of exports from land to sea trade; iii) industries which are more sensitive to shipping costs respond more; iv) a significant amount of trade actually takes place through the new port; etc. Some of the approaches 1 and 2 could be used to help establish some of these points. I find some of the work economic historians do in trying to explain the causes and consequences of particular events are particularly good at this sort of approach (e.g. the work of Engerman and Sokoloff). Process evaluation plays an important role in this – measuring lots of inputs and usage variables, but it is still important to try and spell out a counterfactual and rule out alternative explanations as much as possible.
Approach Four – take a Bayesian approach: once these sorts of studies have been done well for a few countries, they can feed into evaluations in new countries. For example, if I’ve evaluated the impact of building new ports in Liberia and Guinea, I might be justified in having a strong prior that the effect could be quite similar in Sierra Leone – and then use any information about impacts in Sierra Leone in a comparative sense, to update my estimates as I get new information.
Other approaches? In some cases, a variety of other creative approaches might also add additional information or be possible. For example, stock market event studies could be used to measure the value the stock market ascribes to the port based on the unexpected announcement that it will be constructed. I’m not sure how likely it is that this will be unexpected in most cases though. In some cases one might be able to conduct encouragement designs to get firms or individuals to use the new port or new infrastructure, but in most cases it seems likely that the power to do this will be too low.
Note that in most cases none of these approaches will be completely convincing by themselves, but the combination of approaches and a careful narrative and explanation of what the underlying assumptions about counterfactuals are will provide at least some confidence that one is getting a reasonable sense of the overall impact. Another issue to worry about of course is the timing over which effects are measured- some big infrastructure projects such as ports may take years to have their full effects (see Michael Woolcock’s post on this issue).
As the above illustrates, coming up with credible estimates of the impact of the “one big thing” type of development intervention will be hard, but that one can certainly take some steps towards constructing defendable counterfactuals. Since this is not the type of work I typically do or an area I typically work on, I’d love to see suggestions from our readers of papers or reports they think do a particularly good job in assessing impact of “one big thing”.
Join the Conversation