Tom Vogl is an Associate Professor of Economics at UCSD, and the co-editor at the Journal of Development Economics in charge of their new short papers track. We asked him to reflect on the experience so far.
Overview
At the start of 2021, the JDE rolled out a short papers track in the mold of AER: Insights. Papers may have at most 6000 words and at most 5 exhibits. Referee deadlines are tight, but reports may be very short. Decisions are either rejections or conditional acceptances, with no opportunity for major revisions or additional rounds of peer review. Editor in Chief Andrew Foster and I have been handling all of the submissions. Andrew is in charge of desk rejections. I handle most of the submissions that get past his desk, and he sees a few all the way through the review process. We hold short papers to the same quality standard as full-length papers. One way to interpret this statement is that in the long run, we expect short papers to have the same average citation counts as full-length papers.
The new track has had a successful start, with many high-quality submissions. The quality has been so high that I have generally only been able to conditionally accept papers that received unanimous support from the referees, or something extremely close to it. The “no major revisions” policy has turned out to be a real constraint. I have interpreted “major” to mean any revision that requires evaluation by an original referee. So I have had to reject a number of short submissions that might have received “revise and resubmit” decisions as full-length submissions. I am pleased with the collection of papers we have published, but I have been sad to reject a paper on more than one occasion.
Submissions and decisions
From December 2020 to present, we received 275 short submissions and accepted 18. Some papers are still out for review, so these counts are ill-suited for computing an acceptance rate. Of the 258 submissions with first round decisions, we desk rejected 160 (62%), rejected 71 (28%) after review, and conditionally accepted 27 (10%). Among papers that got past the desk, we conditionally accepted 28%.
For comparison, over the same period, 8% of full-length submissions received an R&R or better in the first round of review, or 32% conditional on getting past the desk.
Turnaround time
The average time to first decision was 5.6 weeks. That average includes desk rejections; unfortunately, the publisher does not give me an easy way to condition the average on getting past the desk. I’ll be honest and admit that I had hoped for a bit better, something like 6 weeks conditional on going out to referees. Our unconditional 5.6-week average clearly fell short of that goal. One issue is that Editor in Chief Andrew Foster reads every submission, short or full-length, to assess whether it should go out for review. Andrew works very hard, but the process takes time. He gets more than 30 submissions a week.
Conditional on going out to referees, we do seem to have handled short papers faster than full-length papers. The average time from submission to referee invitation was 3.4 weeks for short papers, compared to 4.5 weeks for full-length papers. And referees averaged 5.4 weeks to return their reports on short papers, compared to 8.6 weeks for full-length papers. That’s already a month saved. Kudos to the 207 reviewers who have submitted such timely reports! Kudos also to folks who declined our invitation quickly and suggested alternate reviewers, both of which help our speed.
Reports from other journals
AER: Insights, the Review of Economics and Statistics, and the Journal of Public Economics also have short paper tracks, so some authors have wondered if they can include reports from a rejection at one of these journals when they submit their short papers to the JDE. The answer is yes, but the old reports typically do not obviate the need for new referees.
It is difficult to fully interpret the old reports without knowledge of who wrote them. For example, a report that says “this paper is well-executed but not enough of an advance” might carry more weight coming from a senior researcher than from a PhD student, given the differences in the breadth of their experiences. The lack of information on the referee’s identity makes our problem very different from that of an AEJ editor who receives reports from an AER rejection.
That said, old reports are informative, and we pass them on to the new referees. That does not necessarily shorten the review time, but it may improve a paper’s chances.
What works well as a short paper
The most successful short submissions have had one or two key findings and no complications. Some authors may think that if a project did not quite work out, it is a good candidate for a short paper. The logic would seem to be that the length of the paper should be proportional to the strength of its conclusions. But complications require more space, not less. Complications also invite more demands from the referees, and the short papers track rules out major revisions.
For example, a compelling difference-in-differences design with a single result is a good fit; a difference-in-differences design with “a bit” of a differential pre-trend is not. In the latter case, the authors need space to explain to their readers how they address the pre-trend. Similarly, a well-executed randomized trial with a precise null result is a good fit; a randomized trial with “a bit” of contamination is not. The imperfection necessitates more robustness checks, more explanation.
Long-to-short conversions, and vice versa
When we started the short papers track, one of our visions was to identify rejected full-length papers that would be suitable as short papers without requiring additional rounds of review. This vision was loosely based on the track records of journals like Science and Nature, which routinely help authors shorten their papers after conditionally accepting them. While we have occasionally succeeded, these long-to-short conversions have been rare.
We generally do not ask authors to lengthen short submissions; lengthening would constitute a major revision, which the short paper track does not allow. We also do not allow authors to submit rejected short papers anew as full-length papers.
Join the Conversation