I once received a referee report for a journal submission that said, “In fact, in my view its contribution to science is negative…” The report continued with comments about how the paper lacked “proper and sound scientific inquiry” and was “…unsuitable for publication pretty much anywhere, I think.” Just in case the four-page assault was not sufficient, the report ended with encouraging the authors to “…move onto the next project.” It was hard to avoid the feeling that the referee was suggesting a career change for us rather than simply giving up on this paper… The paper was subsequently published in the Journal of Health Economics, but the bad taste of receiving that report lingered long afterwards…
So, almost five years later, when I was writing my first referee report of 2018, I decided to sign my review. I put my name on the bottom of my report and informed the editor that I signed my report to the authors. I don’t do New Year’s resolutions, but reading the paper and writing the report over a long weekend, the idea of signing my report felt completely natural and right. I didn’t have to think long and hard about it: this is what I was going to do moving forward.
If I did think about it before putting my name on the bottom, two issues dominated. First, would I do the same thing for papers I disliked as I do for papers I thoroughly enjoy reviewing? I quickly decided that the answer was “Yes.” In fact, while I liked this paper, my review was not particularly easy on the paper, asking them some tough questions. Furthermore, the thought of being able to write a tough review – one that might be interpreted as a likely recommendation for “rejection” by the editor – and not being afraid that the authors know my identity seemed completely desirable. It would make me a more careful, fair, and, perhaps most importantly, thoughtful reviewer, who would stick to the facts rather that hurl insults or act on subjective biases.
Second, in one word: poor-quality reviews. We have all received our share of these, and I increasingly get the feeling, especially with papers regularly exceeding 100 pages with appendices nowadays, that the average referee does not spend enough time on a paper. You don’t have to be dishonest or unprofessional to turn in a slap dash review. Sometimes, you’re simply too busy, the deadline is nearing (or, worse, you’re way late), and you can’t be bothered reading this paper carefully. Nothing worse than a referee saying you should do X, and you have already reported on exactly X on page 22. As a computer scientist put it (whose blog on this topic I cite further below): “Even honest people will produce bad-quality reviews out of negligence, laziness or lack of time because they know they will not be challenged. Putting your name on an assessment forces you to do a decent job.” I personally like that commitment device for myself: if I agree to do something that I am willing to put my name on, then I will do at least as good a job as I would have under the cover of anonymity. But, I do see that reasonable people can disagree on this point – an argument I return to below.
Anonymity is not only precarious, but also a hindrance. For example, everyone has a writing style. For me, this is worse, because, for the past six years, I have a public one. I don’t know if my style is distinct or not, but most of you can probably tell which blog in Development Impact was written by which one of us without having to look at the byline. I can’t suddenly switch that off when I am writing a referee report. Also, I have certain views and preferences about how I’d like to see analysis from, say, RCTs conducted and presented. I write about these in this blog and they regularly come up while refereeing reports. For the authors, it is sometimes not that hard to think about whom Referee #1 might be. Worse, the time spent by authors speculating and referees trying to hide their identities is wasteful. You might avoid referring to your own work when, ironically, you were chosen as a referee exactly for that work.
Like I said, I did not hesitate much to sign my name on that review last week. But, doing a brief Google search on the topic brought about a multitude of views about this practice that ranged from praise and support to hostility and disdain. My first hit about signing referee reports was this piece by Bertrand Meyer, a computer scientist. I like the way Dr. Meyer puts the tradeoffs and tries to resolve them for his own approach. He is not particularly preachy (nor, I sincerely hope, am I) – recognizing that there may be scenarios that would require the referee to make an exception.
Trying to find something from economics, I had the misfortune of ending up on an old #EJMR thread: toxic as usual. It suffers from the same problems referee reports do, namely anonymity, under the cloak of which people are free to be… well, let’s just say “themselves.” But, the thread also made me realize that there may be serious resistance to this practice in the profession. For example, I had not seriously thought of the possibility that some editors may not accept signed referee reports.
The main objection in economics to signing your reviews seems to be opening the possibility of “I scratch your back and you scratch mine” between the referee and the authors. I guess this is possible, but a good editorial board should be able to see through this. As Dr. Meyer points out in his post, I worry much more about competitors attacking the work they’ve been assigned to review than being too nice and trying to collude with the authors (in some sort of a vague, dynamic, long-term game). Races against the clock in economics are not uncommon and the premium for being first is unduly high. So, it seems better to protect against nasty and slapdash than nice and fair. Editors have an important job of choosing referees with specific expertise. I speculate that referees signing their reports will also make editors more accountable – even if it may make their jobs slightly more difficult.
I realize that I take these views as a male researcher in a male-dominated profession, who has spent a fair amount of time in it since graduation. I grant that the decision to write a review as a junior, female researcher that may be critical of a senior and influential professor’s work is tougher. We’re already worried about double standards and discrimination towards certain sub-groups in our profession and now they also have to make themselves explicit targets to some vindictive academic by signing their reviews? Journals don’t have to mandate signing your reviews, even if, in some future time that becomes the default option: you can opt to write an anonymous review. But, if your review is suffering from being uncivil or unprofessional, the editor should be able to send you your review back and tell you to try again. In the example I gave above, I often wondered why the editor chose to share that review with us and whether that referee was ever asked to review for that journal again.
Many other issues arise once you start thinking about this issue. For example, a 2013 post on Crooked Timber discussed the issue of revealing your identity as a referee to a third party. That led to discussions about revealing it to the authors, but after publication rather than during the review process; and further discussions about the differences between reviewing as a panelist for, say, the NIH or a job application rather than for a journal. These are all interesting and complex issues that are worthy of debate.
For the time being, I will continue to sign my reports. It seems strange to me that we have been talking about openness and transparency for data and methods, but we argue the opposite for this issue. The irony that I have to hide my identity for an open and transparent scientific process does not escape me. Nor am I convinced that it is needed…
Your constructive comments, as always, are welcome. It would be especially nice to hear from journal editors. You have the option to be anonymous in the Comments section below, but you’re more than welcome to sign your name…
Join the Conversation