The peer-reviewed publication process is something many researchers go through, whether as authors or referees. But we are not always sure how to deal with various decisions, especially earlier in our careers. So, we decided to ask Lawrence Katz, editor of the Quarterly Journal of Economics, his views on a bunch of things, including some advice for prospective authors and referees alike. He was kind enough to take time from his busy schedule to answer our questions…
Development Impact: Though the Quarterly Journal of Economics (QJE) receives more than 1,000 manuscripts a year; you manage to provide high quality reviewer reports to the authors within six weeks from submission. Could you please tell us about the process a little bit from submission to decision?
Larry Katz: The QJE received 1,430 new submissions in 2011 up from around 1,050 five years ago and from around 400 in the early 1990s when I first started as a QJE editor. We added a fourth editor (Jeremy Stein) this past year (in addition to Robert Barro, Elhanan Helpman, and me) to try to keep up with this rapidly rising inflow. I believe that our low turnaround time, efficient handling of manuscripts, and attempts at providing authors with quality reports and feedback play some role in our large number of submissions and in the high impact factor (citation rate) of papers published in the QJE.
The process by which new submissions are handled at the QJE is straightforward. It is also rather boring, but since you asked …
A new submission is allocated to an appropriate QJE editor on the day of submission by whichever QJE editor logs into our Editorial Express system and first encounters that submission. Once a paper is allocated the assigned editor typically reads the paper within 1 day and decides whether to send the paper out to referees or to “desk reject” it. Since we receive over 1,400 submissions and only publish around 40 to 44 papers a year, we need to be tough in the initial screening and only send a paper out to referees if it has great promise to be a significant contribution. This past year we desk rejected 62 percent of the new submissions (which still means that over 500 were sent to referees). I am the “softy” among QJE editors desk rejecting “only” 47 percent of new submissions vs. 70 percent by my co-editors. I also try to provide authors with some brief feedback on their papers and the rationale for the decision even in the case of desk rejections.
If a paper is viewed as having a serious shot at publication, we typically send it out to 2 to 5 referees. For papers sent out to referees, my median turnaround time is 33 days over the past few years. The key to maintaining reasonable turnaround time is creating norms and expectations among referees that reports should be done in a timely manner (4 weeks is our expected due date) and that the editors will quickly make decisions. The QJE editors monitor the refereeing process and look at each referee report as it comes in rather than waiting for all the reports to have been completed. (I typically log into our Editorial Express system at least a couple times a day to monitor my pending manuscripts and check for new submissions.) Thus, if the initial referee reports received recommend rejection and are convincing, the editor makes a rejection decision without waiting for all the reports.
The hurdle for asking for a revision is quite high at the QJE since we can only publish a small share of the papers sent out to referees. I try to be direct but as polite and constructive as possible in rejection letters. Most papers we reject with referee reports are strong papers that end up being published in other leading journals.
Our revise and resubmit letters try to be as precise as possible in providing authors with a road map to what changes need to be made (and which of the referees’ suggestions should be followed). The expectation is that a responsive, serious, and successful revision is likely to be accepted for publication. The vast majority of papers we ask for revisions do end up being accepted. I know there is nothing more frustrating for authors than being asked to completely redo your paper in a major revision and then being rejected.
Once a revision has been received, we typically send it back to the most relevant subset of referees and then make an up-or-down decision on the first revision. Any further rounds after this “conditional acceptance” involve minor (but essential) revisions to make the paper as reader-friendly and concise as possible and get it into final shape for publication.
The QJE has demonstrated for the past two decades that one can run a major economics journal with reasonable turnaround times and still provide high-quality feedback. The new AEJs are following in this tradition and several field journals (and even the AER) have made progress in improving turnaround times.
DI: Quite a few of our readers are PhD students, young faculty, and young researchers at the World Bank who are submitting a manuscript to the QJE, as well as becoming reviewers, for the first time. What are your pet peeves with submissions? Is there anything that will get an automatic desk reject? Do you have any advice to junior faculty and graduate students contemplating a submission to a top journal? How about reviewers: anything you would like them to avoid? Is it your experience that young referees are overenthusiastic and typically write too much?
LK: Young scholars should NOT submit a paper to a top journal too soon. You should only submit polished work that has been presented several times and incorporated the critical feedback of colleagues and/or advisors. You are simply closing off options if you submit a paper before it is ready. Papers need to be well-written and self-contained. A paper will get desk rejected for sure if it is sloppy (missing references, tables and figures that can’t be deciphered, equations with notation that is not defined, poor and verbose writing, etc.) regardless of the quality of the ideas.
Reviewers should provide a concise summary of the paper they review at the start of their report and then provide a critical but polite evaluation of the paper. The referee report itself should not include an explicit editorial recommendation. That recommendation should be in a separate cover letter to the editor. The more likely you think the paper is to merit a revision the more detailed should be the comments.
The other key advice I have for potential reviewers is that if you are time constrained then it is much better to immediately tell an editor that you can’t do a report in a timely manner than to say yes to doing the report and not come close to meeting the deadline. Furthermore, a brief but timely report for a paper that is a clear reject upon your initial reading is much preferable to a delayed but more detailed report.
DI: There are far fewer papers published on developing countries than on the U.S., especially in top journals. One of the most common laments of development economists in receiving reports back from top journals is to hear that the subject is "not of general interest". Nevertheless, the QJE has thankfully published a decent number of development articles in recent years - any tips or thoughts on what it takes to craft a paper as of general interest?
LK: In my view, the country (or countries) being studied is not that relevant for whether a paper is considered of general interest. Lots of U.S. studies also get rejection letters indicating they are not of general interest. The degree of methodological and substantive innovation and the quality of the ideas and research are what matter. It is reasonable to try to provide readers with some sense in the introduction and conclusion of the broader (but direct) lessons that can be distilled from one’s research. But one should not go off the deep end in doing this.
High-quality research in development economics has proliferated in recent years. In fact, there are now more non-U.S. empirical studies under revision at the QJE than studies focused on U.S. data. The tide is turning.
DI: A common query referees and editors have about some impact evaluations, particularly from developing countries, is about external validity, with the concern being that the findings only apply to one country or sample. But efforts to replicate studies are then met with the complaint that they are not novel enough. Any advice to authors of the two types of papers - what do you look for in making the case for some external validity? What are your thoughts on the circumstances on which the same experiment or study replicated in a different country or setting would be of sufficient interest for a general interest journal?
LK: I think questions of external validity need to be assessed on a case-by-case basis. I believe replication studies are important, but I don’t think it is likely that many pure replication studies will end up getting published in major economics journals unless they generate new lessons (e.g., which policies replicate and which ones don’t or something systematic about what settings do or don’t generate larger treatment effects). Single replication studies just are not that exciting. I hope that the leading field journals will devote more space to convincing replication studies since they are valuable and should get some professional recognition.
DI: There has been quite a bit of discussion in the last year or so about the need to start a trial registry for randomized trials in economics. This would only work if editors of journals provide incentives for people to use it. Any thoughts on the sorts of things you'd like to see in such a registry? Should it include lab experiments as well as field experiments? Do you see publication in journals such as the QJE becoming contingent on registration, as is the case with many medical fields, or a different role (e.g. as a repository for pre-analysis plans) being the main role?
LK: I am sympathetic to creating a registry for randomized field experiments and for laboratory experiments in economics to improve transparency and provide opportunities for sticking to a pre-analysis plan. I would hope the QJE would follow the guidelines set by AEA journals in encouraging such registration of submitted papers. Making one’s data available to other researchers to the extent possible given human subjects concerns is also important.
But I do think that it is not so clear what is a randomized trial vs. a natural experiment. And there are difficulties when some randomized trials are parts of government programs and the eventual researchers are not those that set up the program or the trial itself. We need to remain open to scholars ex post using plausible sources of variation (even from earlier randomized field trials) to provide better answers to important questions even if they were not the ones that motivated the initial policy or field trial. We can’t expect someone to register the Vietnam draft lottery. Still we need to be aware of multiple hypothesis testing issues.
DI: You have a recent paper in the New England Journal of Medicine. This is part of a trend among economists as we have more experimental evidence of social programs on health. However, manuscript lengths, requirements, and templates are substantially different in the biomedical field. What do you think of the publication process in these journals? Should economists be publishing more or less in such journals?
LK: It is a tough slog to get through the medial journal publication process. Thus, any economist trying to submit to a major medical journal needs a collaborator with experience in publishing in medical journals and in responding to the referees at medical journals. I think it is socially productive for the innovative work of economists and other social scientists on health topics to be more widely available to medical researchers and practitioners. More publishing by economists in the leading medical journals is a good way to make that happen.
Thanks for sharing. This is useful.
You might also want to read Robert Moffitt's take on similar issues in the Spring 2011 CSWEP newsletter http://www.aeaweb.org/committees/cswep/newsletters/CSWEP_nsltr_SprSum_2…
done when he was editor of the AER.