Is the second round of refereeing worth it?

At the margin, probably not. So if that’s what you’re looking for you can go back to your coffee. But a neat new paper by Aboozar Hadavand, Daniel Hamermesh, and Wesley Wilson provides not only the answer to this question but a host of interesting data and insights into the refereeing process.

Before we get to their result, lets take a look at this interesting data and estimates on the refereeing process.  First, they talk to the top four economics journal and find out that they averaged 2,424 referee reports in 2018. They take this and extrapolate out with other journals (assuming less for most) and end up with an estimate of 300,000 (!) referee reports in economics per year. They assume that the average report takes 5 hours (based on their personal experience). This adds up to 1.5 million hours per year. Taking the average salary in economics, adjusting it for “value of time in non-market activity”, this comes out to a cost of over $50 million a year.

So, a lot of work goes into refereeing. What Hadavand and co. want to know is does the extensive process of revise and resubmit pay off? But, before we get to how they answer that, one other important piece of motivation. Economics, as we all might know or suspect, takes longer than other disciplines to get a paper into print. Anyone who has submitted to a science or health journal has experienced this, but it turns out this is also true for sociology, psychology and political science. Hadavand and co. find that the median time from submission to publication in a (small) sample of journals from these fields is 17 months – a year less than a sample of (not the top) economics journal. Looking at the tail of the distribution, this comparison is even worse.

Hadavand and co. set out to look at the value of this longer process by examining the impact of multiple rounds of refereeing. They work with submissions to Economic Inquiry (EI), which in 2007 introduced a two-track submission process.  Folks submitting had a choice between a) a fast track, which had a refereed thumbs up/thumbs down decisions (and the option for the authors to edit based on comments), and b) the usual process which could involve multiple rounds.

Hadavand and co. then look at the papers submitted between 2009 and 2018. They collect information on the authors, including the year in which they got their PhD (which allows them to calculate a “PhD-age” at time of submission) and prior citations. They also measure the citations papers get once EI publishes the paper.

The first result is that if you compare time to decision there is no meaningful difference between the first decision in the fast track versus the regular track. This makes sense, since they are both refereed.  But the regular track allows multiple rounds – and here it gets stickier. At the 90^th percentile of the distribution of regular track time to decision, it takes 512 days and this jumps to 640 days at the 95^th percentile.

But fast track papers are (significantly) slightly more likely to be published (0.159 to 0.149).  And yes, as you might suspect, the authors are different.  Looking at the most cited and most senior authors, the fast track authors outrank their regular track counterparts. Fast track papers are more likely to include a female author and less likely to only have one author. So, these are differences that Hadavand and co. will control for in any quality comparison. 

With none of these variables controlled for, fast track papers have significantly more post-publication citations.  However, the controls don’t make this go away – even controlling for these dimensions of author differences, fast track papers get more cites. Perhaps more interesting is that (controlling for author qualities), the one round revise and resubmits do not get significantly different number of cites from multiple rounds of revisions.  

One question one might ask is whether fast track articles had more referring prior to submission. Hadavand and co. try to rule this out by conducting a survey of authors. Responses aren’t great (62 percent) and authors memories are faulty (mine surely is), but as far as they can tell, there isn’t a huge difference across the tracks.  

Now, of course, the decision on which track to use is endogenous. So Hadavand and co. use an IV approach to try and net this out. Here it turns out prior citations are a good prediction, as well as the years post-PhD (they use this both in a continuous sense, but also if you fall in the up-for-tenure range).  With this IV approach, there aren’t significantly higher citations for fast track papers. But the lack of a difference between one round revisions and multiple round revisions in the regular track holds. So the extra round of comments doesn’t seem to get you a better paper – in terms of citations.

Null results are always harder to prove so Hadavand and co. go one step further.  They take a look at the editors at EI and are able to rule out potential editor effects as explaining this.  

All in all, this is an interesting paper. Lots to think about regarding the speed of refereeing beyond the first round, as well as the value of those comments – at the margin (as their title says).