Syndicate content

Add new comment

Submitted by Owen Ozier on
Dear David,

Thanks for that comment!  In fact, I had thought of linking to Registered Reports, but cut it in the interest of brevity.  I'm glad you have linked it here.

Since you are here, could you also comment on two things?

1) How does your initiative grapple with the asymmetry that, for some research questions, a significant finding is of greater importance than an insignificant finding (even a precise zero)?  Journals with limited space (and busy readers with limited attention) might need to condition on the actual findings, not just the analytical plan?  In other words, how many headlines begin with the phrase, “researchers FAIL TO discover new…” ?  It seems to me that the answer could lie in whether a journal’s commitment, based on the pre-analysis plan, precludes submission to other outlets.  (In economics, the norm is that a manuscript cannot simultaneously be under consideration at multiple journals.) If, for these journals, there was a multilateral agreement that this was allowed, you could get a pre-commitment that this journal will publish your study based on the plan, and then you are allowed to submit to top-tier outlets if you think you have a shot at one – i.e. if the finding is exciting, overturning priors, etc.  If it doesn’t pan out, this journal will publish whatever short report you want to submit with minimal editorial cost.  Is this the direction to be headed?

2) How do we think about imprecise zeros?  A lot of the file drawer problem may be that the zero is imprecise: the survey instrument ended up very noisy, sample size ended up smaller than anticipated, attrition is high, compliance is low, the partner organization didn’t implement well, etc.  A journal editor might think that either a precise zero or a rejected null is interesting, but a poorly-implemented program or a data collection disaster isn’t.  (And isn’t that informative for meta-analyses either.)