Syndicate content

Do randomized trials work?

I'm a fan of randomized trials for aid projects: they provide credible information about the true effects of the intervention. Other evaluation methods struggle - what if the project goes badly but other things in the economy are going well? You might get apparently good results and expand a bad idea. Or vice versa.

But I've been reading up on a symposium in the Journal of Economic Perspectives from 1995 (JStor or AEA membership necessary). Nobel laureate James Heckman and Jeffrey Smith put the counter-argument:

  • Randomized trials, unless very ambitious, don't allow researchers to estimate structural parameters such as the price elasticity of demand, or the labour response to a tax cut.
  • Alternative econometric methods can control for sample selection biases, providing sufficiently rich data are available.
  • Randomized trials can cause 'randomization bias', where the willingness of partners to participate is altered by the fact that the trial will be randomized.
  • Randomized trials do not necessarily provide information about WHY a policy succeeded or failed.

I found Heckman & Smith's paper interesting but not compelling - bearing in mind that Heckman is a chief architect of econometric techniques designed for use in a world where experiments are not possible. It's a useful reminder that randomized trials are not perfect, and that we should always look for new and better methods of evaluation. But I've not changed my mind that these trials are appropriate far more often than aid agencies are currently using them.

Add new comment