We promised some time ago to review the recent working paper by Pritchett and Sandefur on external validity, and the title of this post is the main take-away for me: my name is Berk Özler and I agree with this specific message. However, while I’d like to say that there is much more here, I am afraid that I, personally, did not find more to write home about...
A common critique of many impact evaluations, including those using both experimental and quasi-experimental methods, is that of external validity – how well do findings from one setting export to another? This is especially the case for studies done on relatively small samples, although as I have ranted before, there appears to be a double standard in this critique when compared to both other disciplines in economics and to other development literature.
I’ve been reading Evidence-based policy: a practical guide to doing it better by Nancy Cartwright and Jeremy Hardle. The book is about how one should go about using existing evidence to move from “it works there” to “it will work here”. I was struck by their critique of external validity as it is typically discussed.
In recent conversations on research, I’ve noticed that we often get confused when discussing the placebo effect. The mere fact of positive change in a control group administered a placebo does not imply a placebo effect – the change could be due to simple regression to the mean.
Well I’m writing this on Election Day evening here in the U.S., and am rather consumed by the events at hand.
There is a minor buzz this week in Twitter and the development economics blogosphere about a paper (posted on the CSAE 2012 Conference website) that discusses a double blind experiment of providing different seeds of cowpeas to farmers in Tanzania.
A core concern for any impact evaluation is the degree to which its findings can be generalized to other settings and contexts, i.e. its “external validity”. But of course external validity concerns are not unique to economic policy evaluation; in fact they are present (implicitly or explicitly) in any empirical research with prescriptive implications.
For the World AIDS Day, there is a sign at the World Bank that states that taking ARVs reduces rate of HIV transmission by 96%. If this was last year, a sign somewhere may well have read “A cheap microbicidal gel that women can use up to 12 hours before sexual intercourse reduces HIV infection risk by more than half – when used consistently.” Well, sadly, it turns out, so much for that.
When done well, randomized experiments at least provide internal validity – they tell us the average impact of a particular intervention in a particular location with a particular sample at a particular point in time. Of course we would then like to use these results to predict how the same intervention would work in other locations or with other groups or in other time periods.