Syndicate content

evaluation methods

Are we over-investing in baselines?

Alaka Holla's picture

 
When I was in second grade, I was in a Catholic school, and we had to buy the pencils and pens that we used at school from a supply closet. One day I felt like getting new pencils, so I stood in line when the supply closet was open and asked for two. Before reaching for the pencils, the person who operated the supply closet, Sister Evangelista, told me a story about her time volunteering in Haiti, how the children she taught there used to scramble about in garbage heaps looking for discarded pieces of wood, charcoal, and wire so that they could make their own pencils. I left the closet that day without any pencils and with a permanent sense of guilt when buying new school supplies.
 
I now feel the same way about baseline data. Most of the variables I have ever collected – maybe even 80 percent – sit unused, while only a small minority make it to any tables or graphs. Given the length of most surveys in low- and middle-income countries, I suspect that I am not alone in this. I know that baselines can be useful for evaluations and beyond (see this blog by David McKenzie on whether balance tests are necessary for evaluations and this one by Dave Evans for suggestions and examples of how baseline data can be better used). But do we really need to spend so much time and resources on them?  
 

Seeking nimble plumbers

Alaka Holla's picture
Sometimes (maybe too many times), I come across an evaluation with middling or null results accompanied by a disclaimer that implementation didn’t go as planned and that results should be interpreted in that light. What can we learn from these evaluations? Would results have been better had implementation gone well? Or even if implementation had gone just fine, was the intervention the right solution for the problem? It’s hard to say, if we think of program success has a product of both implementation and a program that is right for the problem.

Evaluating an Argentine regional tourism policy using synthetic controls: tan linda que enamora?

David McKenzie's picture
In 2003, the Argentine province of Salta launched a new tourism development policy with the explicit objective of boosting regional development. This included improving tourism and transport infrastructure, restoring historical and cultural heritage areas, tax credits for the construction and remodeling of hotels, and a major promotion campaign at the national and international levels.

Evaluate before you leap -- volunteers needed!

Markus Goldstein's picture

It’s been a year since we started the Development Impact blog, and I thought I would use the one year anniversary to focus on one of the classic papers in impact evaluation.    This paper (gated version here, ungated version here) is by Gordon Smith and Jill Pell and appeared in the BMJ back in 2003.

Can we trust shoestring evaluations?

Martin Ravallion's picture

There is much demand from practitioners for “shoestring methods” of impact evaluation—sometimes called “quick and dirty methods.” These methods try to bypass some costly element in the typical impact evaluation. Probably the thing that practitioners would most like to avoid is the need for baseline data collected prior to the intervention. Imagine how much more we could learn about development impact if we did not need baseline data!

Strategies for Evaluating the Impact of Big Infrastructure Projects: How can we tell if one big thing works?

David McKenzie's picture

One of the interesting discussions I had this last week was with a World Bank consultant trying to think about how to evaluate the impact of large-scale infrastructure projects. Forming a counterfactual is very difficult in many of these cases, and so the question is what one could think of doing. Since I get asked similar types of questions reasonably regularly, I thought I’d share my thoughts on this issue, and see whether anyone has good examples to share.