Syndicate content

pre-analysis plans

Towards policy irrelevance? Thoughts on the experimental arms race and Chris Blattman’s predictions

David McKenzie's picture

Chris Blattman posted an excellent (and surprisingly viral) post yesterday with the title “why I worry experimental social science is headed in the wrong direction”. I wanted to share my thoughts on his predictions.
He writes:
Take experiments. Every year the technical bar gets raised. Some days my field feels like an arms race to make each experiment more thorough and technically impressive, with more and more attention to formal theories, structural models, pre-analysis plans, and (most recently) multiple hypothesis testing. The list goes on. In part we push because want to do better work. Plus, how else to get published in the best places and earn the respect of your peers?
It seems to me that all of this is pushing social scientists to produce better quality experiments and more accurate answers. But it’s also raising the size and cost and time of any one experiment.

Preregistration of studies to avoid fishing and allow transparent discovery

Berk Ozler's picture
The demand for pre-analysis plans that are registered at a public site prior available for all consumers to be able to examine has recently increased in social sciences, leading to the establishment of several social science registries. David recently included a link to Ben Olken’s JEP paper on pre-analysis plans in Economics. I recently came across a paper by Humphreys, de la Sierra, and van der Windt (HSW hereon) that proposes a comprehensive nonbinding registration of research. The authors end up agreeing on a number of issues with Ben, but still end up favoring a very detailed pre-analysis plan. As they also report on a mock reporting exercise and I am also in the midst of writing a paper that utilized a pre-analysis plan struggling with some of the difficulties identified in this paper, I thought I’d link to it a quickly summarize it before ending the post with a few of my own thoughts.

The Impact of Vocational Training for the Unemployed in Turkey: an inside look at my latest paper

David McKenzie's picture
My latest working paper (joint with Sarojini Hirschleifer, Rita Almeida and Cristobal Ridao-Cano) presents results from an impact evaluation of a large-scale vocational training program for the unemployed in Turkey. I thought I’d briefly summarize the study, and then discuss a few aspects that may be of more general interest.

The study

Creativity vs. fishing for results in scientific research

Berk Ozler's picture
One of my favorite bloggers, Andrew Gelman, has a piece in Slate.com in which he uses a psychology paper that purported to show women are more likely to wear red or pink when they are most fertile as an example of the ‘scientific mass production of spurious statistical significance.’ Here is an excerpt:

Does Angus Deaton worry too much about wolves eating his t-values?

David McKenzie's picture

In his latest Letter from America in the Royal Economic Society’s newsletter, Angus Deaton says “your wolf is interfering with my t-value” (the title refers in part to regulations on hunting wolves in the American West) and talks about excessive regulation with NIH grants, and his concerns with the move towards trial registries:

A pre-analysis plan checklist

David McKenzie's picture

A pre-analysis plan is a step-by-step plan setting out how a researcher will analyze data which is written in advance of them seeing this data (and ideally before collecting it in cases where the researcher is collecting the data). They are recently starting to become popular in the context of randomized experiments, with Casey et al. and Finkelstein et al.’s recent papers in the QJE both using them.