Syndicate content

Add new comment

I like this comment as it reveals that the whole movement for basing policy on "rigorous evidence" (by which was meant either RCTs or evidence that people liked the identification of) is on its last legs.

First, suppose an RCT has been done on impact of job training on earnings in five American cities and not yours, say my home town of Boise Idaho. Suppose there are some observables that are available in the five cities and that hence I can predict from the RCT the likely causal impact in Boise. One can do that, but at this stage all of the propaganda of "rigorous" is lost. There is nothing defensibly more "rigorous" about this evidence than any other evidence that could be deployed. That is, I might have an OLS estimate of the association of wages and actual job training from implementation in Boise. There is no sense in which the extrapolation of evidence from an RCT elsewhere is more "rigorous" than OLS in Boise. That is, we know the internal validity issues that "correlation is not causation" for OLS but we also know the problems with external validity. So this is a tradeoff between two pieces of non-rigorous evidence so all rhetoric about "using rigorous evidence" is now irrelevant as in the proposed use the RCT evidence isn't rigorous.

(see Pritchett and Sandefur 2013 for empirical examples where it appears the external validity problems of variation across context are much worse than internal validity problems with causal identification).

Second, it is worth noting how wildly at odds what is being discussed is from most potential development applications. This is extrapolating from Riverside California to, say, Boise Idaho. But what about Riverside California to Cali Colombia? Or Ankara Turkey? or Nairobi Kenya? We would have to suppose that the variation in causal impact across contexts is mostly? primarily? exclusively? captured by measures that are measured in the RCT sites and that these measures of also comparable to the measures in policy application contexts? But without a complete, coherent, theoretically sound and empirically validated model that provides the "invariance laws" these adjustments are stabs in the dark.

Take the example of job training. Suppose that job training is more effective in US cities with lower measured unemployment rates because, it just so happens, given the US context these proxy across cities/regions for strong labor demand. But the "unemployment rate" may well be raised by transfer programs that allow for greater search. So it could be a poor city has a lower measured unemployment even with strength of labor demand. In this case not only would the casual impact lack external validity but the adjustments to external validity would lack external validity as lower unemployment could be associated with less impact in a city in a poor country rather than bigger impact as the adjustment would suggest. Maybe my example is false--but in extrapolating results across contexts no one knows if it is or not and the evidence doesn't tell us.

So in nearly all development applications we are in exactly the position Jed suggests in which "little can be done" with existing RCT estimates.