Andrew Gelman has a post from last week that discusses the value of preregistration of studies as being akin to the value of random sampling and RCTs that allow you to make inferences without relying on untestable assumptions. His argument, which is nicely described in this paper, is that we don’t need to assume nefarious practices by study authors, such as specification searching, selective reporting, etc. to worry about the p-value reported in the paper we’re reading being correct.
The demand for pre-analysis plans that are registered at a public site prior available for all consumers to be able to examine has recently increased in social sciences, leading to the establishment of several social science registries. David recently included a link to Ben Olken’s JEP paper on pre-analysis plans in Economics. I recently came across a paper by Humphreys, de la Sierra, and van der Windt (HSW hereon) that proposes a comprehensive nonbinding registration of research. The authors end up agreeing on a number of issues with Ben, but still end up favoring a very detailed pre-analysis plan. As they also report on a mock reporting exercise and I am also in the midst of writing a paper that utilized a pre-analysis plan struggling with some of the difficulties identified in this paper, I thought I’d link to it a quickly summarize it before ending the post with a few of my own thoughts.