When I was in second grade, I was in a Catholic school, and we had to buy the pencils and pens that we used at school from a supply closet. One day I felt like getting new pencils, so I stood in line when the supply closet was open and asked for two. Before reaching for the pencils, the person who operated the supply closet, Sister Evangelista, told me a story about her time volunteering in Haiti, how the children she taught there used to scramble about in garbage heaps looking for discarded pieces of wood, charcoal, and wire so that they could make their own pencils. I left the closet that day without any pencils and with a permanent sense of guilt when buying new school supplies.
I now feel the same way about baseline data. Most of the variables I have ever collected – maybe even 80 percent – sit unused, while only a small minority make it to any tables or graphs. Given the length of most surveys in low- and middle-income countries, I suspect that I am not alone in this. I know that baselines can be useful for evaluations and beyond (see this blog by David McKenzie on whether balance tests are necessary for evaluations and this one by Dave Evans for suggestions and examples of how baseline data can be better used). But do we really need to spend so much time and resources on them?
There is much demand from practitioners for “shoestring methods” of impact evaluation—sometimes called “quick and dirty methods.” These methods try to bypass some costly element in the typical impact evaluation. Probably the thing that practitioners would most like to avoid is the need for baseline data collected prior to the intervention. Imagine how much more we could learn about development impact if we did not need baseline data!
One of the interesting discussions I had this last week was with a World Bank consultant trying to think about how to evaluate the impact of large-scale infrastructure projects. Forming a counterfactual is very difficult in many of these cases, and so the question is what one could think of doing. Since I get asked similar types of questions reasonably regularly, I thought I’d share my thoughts on this issue, and see whether anyone has good examples to share.
- evaluation methods