The State of Development Journals 2017: Quality, Acceptance Rates, and Review Times

|

This page in:

I recently became a co-editor at the World Bank Economic Review, and was surprised to learn how low the acceptance rate is for submitted papers. The American Economic Review and other AEA journals such as the AEJ Applied publish annual editor reports in which key information on acceptance rates and review times are made publicly available, but this information is not there for development economics journals. Since I think many of our readers would be interested in this information, I contacted the editors at different journals, and thanks to their cooperation, can share some key information.

In deciding which journal to submit a paper to, three key questions that are often on the minds of authors are 1) Is this a good quality, high visibility journal to publish my work? 2) What are the chances of my paper being accepted?, and 3) How long will they take to review my work? I address each in turn.
  1. Is this a good quality, high visibility journal to publish my work?
While most research is first released as a working paper, journal publication still plays a key role in research production and dissemination in at least two ways. First, hopefully the peer review process helps improve the quality of the paper through the suggestions of the editor and referees. Second, publication acts as a signal of paper quality, which is important for readers seeking to sift through the vast amount of research being produced, and for career concerns like promotions.
I consider several metrics of journal quality:
  • The best known is the impact factor of the journal, which is available as both a 2-year and 5-year factor. The standard impact factor is the mean number of citations in the last year of papers published in the journal in the past 2 years, while the 5-year is the mean number of cites in the last year of papers published in the last 5.
  • A second measure comes from RePec’s journal rankings which takes into account article downloads and abstract views in addition to citations.
Table 1 shows these data, sorted by impact factor.  

Note how low the impact factors are for all journals, with an average of at most 1-2 citations per article per year. There are several problems with these, including the fact that the means can be skewed by a few outliers, and that the long publication lag in economics makes it take a long time for citations to show up. I therefore took the 2015 issues of each journal, and looked up the Google Scholar citations of each paper published. Figure 1 then gives a boxplot of the data, sorted by mean number of citations.

Figure 1: Boxplot of citations as of February 2017 of articles published in 2015

(notes: I excluded papers and proceedings and the Journal of African Economics conference symposiums, the AER is only for development papers published there and is included as a benchmark, and I took 6 out of 12 issues of World Development as a sample given the large number of papers there).

Three things are key from this figure. First, the number of citations are much larger than the impact figures would suggest for many journals: e.g. the WBER has a mean of 29 and median of 12 citations per article, compared to the impact factor of 1.18. Second, comparing journals based just on mean citations seems an incredibly poor way of assessing the typical quality. The medians can differ substantially from the means, the ranges can be large, and the distributions are skewed. Third, there is a lot of overlap in the distributions (a regression of citations in development journals on journal dummies has an R2 of 0.18).

2. What are the chances of a paper being accepted?
Table 2 reports the number of submissions received by each journal in 2016, the number of papers published, and the acceptance rate. By way of comparison, I also report these for the AER and AEJ applied. I found three things notable in this table: (i) The large number of papers many journals receive as submissions each year: more than a paper every day of the year for the WBER, EDCC and Journal of African Economics, more than 1,000 papers for the JDS and JDE, and more than 2,300 for World Development! This shows the enormous job facing editors, who have a huge number of papers to deal with. (ii) the acceptance rates at the top development journals are as low (5 to 7 percent) as for the AER and AEJ applied. While this doesn’t control for the quality of submissions, I still found the rates lower than I would have thought; (iii) I hadn’t realized before quite the extent of variation in the number of articles published by journals             

(note: in data provided Feb 25, 2017: EDCC had acceptance rate of 9% in 2015, and 8.2% for papers submitted in 2016 for which final decisions had been made).

3. How long does the review process take at each journal?
Table 3 reports data on how long journals take deciding on papers. This information was the most difficult to obtain from many journals, and it appears that a number of journals do not carefully track standardized metrics on these. The first piece of information I requested was the desk rejection rate. Given the large number of submissions and low overall acceptance rates at many journals, desk rejections are an efficient way to screen out many papers and render quick decisions on them. We see the desk rejection rates at the top development journals are more than 50 percent, and are actually higher than at the AER and AEJ applied. Using these desk rejection rates, I calculate the approximate acceptance rate conditional on going to referees in the second column – you can see that even if you pass the desk rejection stage, less than one-third of papers sent to referees are accepted at many journals.

(note the table above was updated Feb 25, 2017, to reflect new data provided by EDCC).

Secondly, I wanted to see how long it takes to hear back from journals conditional on them sending papers to referees. The AEA journals report the percent of papers refereed in 3 months and within 6 months, with a goal that most papers should be refereed within 3 months. Few of the development journals appear to track these statistics, and it is only the journals with fewer submissions whose editorial assistants were able to easily calculate these. Another statistic many journals report is average time to acceptance for accepted papers. However, it is not clear that journals are measuring this the same way – I think some journals include only the time the paper is at the journal and not time spent with authors doing revisions, while others include the full time from first submission to final decision. Nevertheless, they do show how slow the process can be by time you go through revisions and refereeing-  almost two years at the AER, and a year or more at some development journals.

Of course these outcomes depend heavily on the referees, in addition to what the editors do. I estimate that the journals for which I have data sent 1,960 papers to referees in 2016, which given 2-3 referees per paper, is perhaps 5,000 referee reports! And this doesn’t even include revisions. So pity the poor editors who have to chase up that many reports, and please do your part to referee on time.

Caveats
There are a number of issues that should be kept in mind when considering the above data:
  1. A number of the journals have recently changed editors, and so the statistics here may reflect business under a former editorial team. Indeed, several of the new editors expressed their desires to improve the refereeing and decision process under their leadership. There was even a request to make this an annual state of development journals so progress could be shown.
  2. Another issue which I haven’t addressed is the time between acceptance and publication in a numbered volume. In some of the younger journals this gap is very small, whereas it can be a year or more at other development journals with publication backlogs. This matters when considering citation rates since publication, since a journal which accepts and publishes an article quickly has less time for it to gather citations than a journal which takes a long time to review the paper and then has the paper forthcoming for over a year before it is published.
Was this useful, what else would you like to see if we do another edition in a year?
I hope this is useful for our readers – perhaps in making you feel better about a recent rejection, happier about an R&R or acceptance, or in helping decide where to submit. I would love your feedback on whether this is the case, and what could be done to make it even more useful if we do this again in the future. Are there journals I missed, other statistics you would like to know, or anything else?

 

Authors

David McKenzie

Lead Economist, Development Research Group, World Bank

Annette Brown
February 21, 2017

David, this is very useful and interesting. Thanks! I especially appreciated your analysis on the impact factors. One thing to note is that while your analysis covers development journals in the social sciences, there is also a huge amount of research about development that is published in the health journals. When we looked for the top ten journals publishing impact evaluations of interventions implemented in L&MICs, we found that half of the top ten are health journals. See the blog post titled "What's first for replication is what's next for 3ie's replication programme" on the 3ieimpact.org website for the full list and our methodology. The publication practices of the health journals are very different and result in very different impact factors.

Jessica Hoel
February 21, 2017

David, thank you for this information! A couple other journals to add to your list: American Journal of Agricultural Economics, Agricultural Economics, and QJE. If you're interested in wading into journals for lab-in-the-field + field experiment combos, AER, AEJ:Applied, and JDE are clear top journals, but I would love to know the stats for the Journal of Economic Behavior and Organization also. They published the special SEEDEC issue a few years ago and have published some nice pieces since.

Anonymous
February 21, 2017

Great piece! Would love to have statistics for Environment and Development Economics also included in future iterations.

Guera Romo
February 21, 2017

Characteristics of a paper that suffers desk rejection would be useful

Thomas Sohnesen
February 21, 2017

Thanks David.very usefull. Review of income and wealth, social indicators research and journal of economic inequality could be journals to add.

Hai-Anh Dang
February 21, 2017

Hi David, this is a wonderful piece! I really learnt much from this and your earlier pieces that interviewed journal editors. And I have a question. Given the different submission times, and the fact that not every one of our papers may have good chance at the top journals, and we have a limited number of papers to submit, do you have any thoughts on a good strategy for submission that minimizes waiting times but maximize outcomes for us young researchers? Say, if I have a paper that may only have a 15% chance of getting a R&R at the JDE or WBER (in my subjective opinion), should I still try it there? Or should I try the other journals? Should my decision change if this chance is larger or smaller?
Thanks, and this is a very nice piece again.

David McKenzie
February 25, 2017

Two more comments received from journal editors by email, which I'm recording here, in part so that if I or someone does a follow-up round next year, all comments are in one place:
1) A couple of journals noted they had experienced a large increase in submissions over the past 5 years or so, with many of these coming from developing country scholars. There was anecdotal evidence that many of these were desk rejects, and so if these were excluded, acceptance rates may look higher. So in a future edition, might be worth seeing if there is a breakdown by country of submission, and if there are any acceptance rates etc. by this breakdown.
2) Similarly they wondered how the presence or absence of submission fees affected the number of submissions, especially from developing countries. I don't think any of the development journals charge these fees, but something to measure in the next iteration.

Jann Goedecke
February 24, 2017

Excellent work David, thanks for the time you invested. You really collected key statistics. One journal I'd like to suggest for future consideration is Development Policy Review. It features lots of development economists' work.

Emmanuel
October 17, 2017

This is very helpful.Thanks a lot.

Eugenie Birch
April 17, 2018

Superb and article. You might comment on the impact factor metric itself, noting that it is an invention of the publishing companies that count the references within that same journal within a strict time limit. Not only is this self-seeking but also neglects citations in other journals.
If you are going to add other journals to your assessment, here are a couple: Environment and Urbanization, International Journal of Urban and Regional Research.

Swagatika Nanda
September 20, 2018

Great piece of Information. I have experienced the rate of acceptance for the paper related to international finance is less.

Alexander Ayertey Odonkor
February 23, 2019

I find the statistics on this subject helpful. This piece is informative and timely.

Karuna
May 02, 2019

Very useful information. Thank you. Any insights on whether the author score also be a criteria for acceptance by the journals?