Attend Spring Meetings on Development topics from Apr 17-21. Comment and engage with experts. Calendar of Events

Syndicate content

Add new comment

Hi Lant, thanks so much for your thoughts. I agree that extending policy lessons from setting A to setting B requires structural assumptions (so they better be carefully considered) and likely adjustments through quasi-experimental methods. There is just no way around this and the blind recommendation of "RCT-rigorous" evidence from setting A to B is terrible practice. Fortunately in my daily experience I don't see this that often. If anything I fear we may be evaluating too much - i.e. re-evaluating the whole kitty in A' (a setting highly similar to A), instead of thinking carefully about how the lessons from A translate to A'.

It reminds me of the story of the government official in, say, Zambia, who says "don't tell me about evidence from Rwanda - I only want to know what will work here". And in many senses this official is correct and acting in the best interests of her constituents and the state budget.

I would still posit there is room for RCTs as one tool she should consider for situated policy learning - in some settings it may be the best choice, in others it may be entirely inapplicable, and perhaps for some questions the "OLS" would yield the closest approximation (and would also be the cheapest approach!). We truly need to systematically think about when, where, and how we recommend a full-blown prospective RCT evaluation versus other approaches. And this decision need include the necessary resources the study would command as well the  existing evidence base for the question asked.

Relatively small RCTs are particularly good for illuminating previously untested interventions/technologies assuming that SUTVA is not violated, etc. RCTs are particularly bad if we care about the performance of a whole system while intervening in a part of the system with something like information that can easily contaminate controls.

So I am quite sympathetic to your view, but perhaps don't lean all the way in your direction. For me, RCTs belong in the toolkit. But if it's the only hammer we employ then we are doing development a great disservice.