Peer review is at the heart of academia; by having papers evaluated by other, anonymous researchers, journals have a great weapon for preventing nonsense from being published. It certainly cannot always prevent bad research from becoming part of scientific knowledge, as the recent replication crises show, but its merits cannot be understated. There are, however, also some problems in the way peer review is implemented that can lead to frustration. First there is the criterium that peer review needs to be double blind. Second there is the problem that each researcher has his own idea of what the norms in a field are and what constitutes good research. Both I believe can be addressed by changing the peer review system.
Anonymity
It may seem strange that I think anonymity of the peer review process is a problem. But there is a number of reasons. The idea behind anonymity was that there would be no reviewer bias: a reviewer would not and could not judge the quality of the paper based on who had written it. That is obviously important, because one would not want some sort of nepotism to take hold of academia. One of the problems is, particularly in modern days, that double blind is an illusion: it is virtually impossible to achieve.
One of the reasons is that most academics share their work either on conferences or online before submitting it. So by that stage a lot of people on an editorial board of a journal will either have heard of the research, or they can simply google it. Which means that reviewers typically know who’s work they’re reading. Another reason is that journals now frequently demand that what they call unnecessary self-citations are avoided, some even don’t want any form of self-citation. But this is obviously quite ridiculous. For one, reviewers might get the idea that the authors have not done their work adequately, because they don’t cite what could be important work. For another, when an established research cannot cite his or her own work, that will leave a huge gap in the paper, and any reviewer will immediately pick up on that. In other words, it actually makes anonymity harder to achieve.
The authors of course do not know who their reviewers are, but who cares about that. Peer review right now is completely asynchronic: a reviewer provides comments and the author(s) has to figure out what s/he might have meant by those comments. Review comments are just as much for the editor of the journal – showing that the reviewer has done his/her job well – as for the author(s) of the paper. There is no adequate recipient design. Misunderstandings and disagreements cannot be resolved. Moreover, as everyone knows, getting anonymous feedback is always more frustrating than regular feedback. As with the Internet, it just feels more insulting when some anonymous person comes by to criticize one’s work and ideas.
Public discussion
Seeing as anonymity is both not achievable and does not necessarily benefit the quality of a paper, what could be advantages of doing it in the open? The best reason is that it allows for discussion. All my writing has improved not simply by getting comments, but by discussing those comments with the people who gave them. By discussing a paper the author(s) gets a chance to understand the reason for the reviewers feedback, which is far more beneficial than simply receiving feedback. I got lucky at some point where one of my reviewers told me s/he was the reviewer: not only did I take the comments less personal (which we just tend to do), but we also got to discuss them which gave me a better idea of how I could address them.
And sure, by getting rid of anonymity we would run the risk of peer review being biased, but we’d be fooling ourselves if we think it isn’t already. As long as there are two to three reviewers for each paper, that should even out. Editors can always look for who they think will be a critical opponent.
Guidelines
My second beef with peer review in its current state is that there are no fixed guidelines: it’s all up to the individual reviewers norms and ideas about what constitutes good academic practice. This can be highly frustrating, because it means that what changes a author(s) has to make depends in part on the luck of the draw. And of course, that will always be the case. Getting the right reviewers might make the difference between a revise & resubmit and a flat-out rejection. But it strikes me as incredibly odd that author A will have to use certain terminology whereas author B who has submitted to the same journal is explicitly told to not use that terminology.
This is a problem that is even harder to address than anonymity, because unlike the judicial system, academia does not work with precedents. Scientific norms and ideas are constantly evolving, which means that just because author A should use some terminology, that does not automatically mean that author B should also be allowed to use it. And we cannot demand that editors keep track of this either; they cannot be expected to know all the comments that have been given to all papers published in their journal.
But allowing for discussion between author(s) and reviewers would at least provide the opportunity for the author to explain why s/he beliefs that the terminology should be allowed (or should not be allowed). In the way we do peer review now, the author can decide to not comply with reviewer comments and explain that in a letter to the editor. But without a chance to adequately defend the argument, one always runs the risk of being overruled. In general, when a reviewer says you should make a change, you have to have a really good reason to not make that change. If the author can at least discuss the comments with the reviewer, they might come to some form of understanding.
Improvement
So in general I think the peer review process would highly benefit from a system of discussion instead of asynchronous feedback. Initially reviewers can still simply provide their comments only to the editor, with a recommendation of reject, revise & resubmit, or accept. But as soon as the editor has decided that a manuscript is not to be rejected, the author(s) should be given a chance to discuss the comments with the reviewers. This doesn’t have to take a whole lot of work: most comments are pretty clear and an author(s) will generally be happy to accommodate a lot of them. But it would definitely provide a chance to further improve the process, and we can stop fooling ourselves that there is such a thing as double blind peer review.
Anonymity
It may seem strange that I think anonymity of the peer review process is a problem. But there is a number of reasons. The idea behind anonymity was that there would be no reviewer bias: a reviewer would not and could not judge the quality of the paper based on who had written it. That is obviously important, because one would not want some sort of nepotism to take hold of academia. One of the problems is, particularly in modern days, that double blind is an illusion: it is virtually impossible to achieve.
One of the reasons is that most academics share their work either on conferences or online before submitting it. So by that stage a lot of people on an editorial board of a journal will either have heard of the research, or they can simply google it. Which means that reviewers typically know who’s work they’re reading. Another reason is that journals now frequently demand that what they call unnecessary self-citations are avoided, some even don’t want any form of self-citation. But this is obviously quite ridiculous. For one, reviewers might get the idea that the authors have not done their work adequately, because they don’t cite what could be important work. For another, when an established research cannot cite his or her own work, that will leave a huge gap in the paper, and any reviewer will immediately pick up on that. In other words, it actually makes anonymity harder to achieve.
The authors of course do not know who their reviewers are, but who cares about that. Peer review right now is completely asynchronic: a reviewer provides comments and the author(s) has to figure out what s/he might have meant by those comments. Review comments are just as much for the editor of the journal – showing that the reviewer has done his/her job well – as for the author(s) of the paper. There is no adequate recipient design. Misunderstandings and disagreements cannot be resolved. Moreover, as everyone knows, getting anonymous feedback is always more frustrating than regular feedback. As with the Internet, it just feels more insulting when some anonymous person comes by to criticize one’s work and ideas.
Public discussion
Seeing as anonymity is both not achievable and does not necessarily benefit the quality of a paper, what could be advantages of doing it in the open? The best reason is that it allows for discussion. All my writing has improved not simply by getting comments, but by discussing those comments with the people who gave them. By discussing a paper the author(s) gets a chance to understand the reason for the reviewers feedback, which is far more beneficial than simply receiving feedback. I got lucky at some point where one of my reviewers told me s/he was the reviewer: not only did I take the comments less personal (which we just tend to do), but we also got to discuss them which gave me a better idea of how I could address them.
And sure, by getting rid of anonymity we would run the risk of peer review being biased, but we’d be fooling ourselves if we think it isn’t already. As long as there are two to three reviewers for each paper, that should even out. Editors can always look for who they think will be a critical opponent.
Guidelines
My second beef with peer review in its current state is that there are no fixed guidelines: it’s all up to the individual reviewers norms and ideas about what constitutes good academic practice. This can be highly frustrating, because it means that what changes a author(s) has to make depends in part on the luck of the draw. And of course, that will always be the case. Getting the right reviewers might make the difference between a revise & resubmit and a flat-out rejection. But it strikes me as incredibly odd that author A will have to use certain terminology whereas author B who has submitted to the same journal is explicitly told to not use that terminology.
This is a problem that is even harder to address than anonymity, because unlike the judicial system, academia does not work with precedents. Scientific norms and ideas are constantly evolving, which means that just because author A should use some terminology, that does not automatically mean that author B should also be allowed to use it. And we cannot demand that editors keep track of this either; they cannot be expected to know all the comments that have been given to all papers published in their journal.
But allowing for discussion between author(s) and reviewers would at least provide the opportunity for the author to explain why s/he beliefs that the terminology should be allowed (or should not be allowed). In the way we do peer review now, the author can decide to not comply with reviewer comments and explain that in a letter to the editor. But without a chance to adequately defend the argument, one always runs the risk of being overruled. In general, when a reviewer says you should make a change, you have to have a really good reason to not make that change. If the author can at least discuss the comments with the reviewer, they might come to some form of understanding.
Improvement
So in general I think the peer review process would highly benefit from a system of discussion instead of asynchronous feedback. Initially reviewers can still simply provide their comments only to the editor, with a recommendation of reject, revise & resubmit, or accept. But as soon as the editor has decided that a manuscript is not to be rejected, the author(s) should be given a chance to discuss the comments with the reviewers. This doesn’t have to take a whole lot of work: most comments are pretty clear and an author(s) will generally be happy to accommodate a lot of them. But it would definitely provide a chance to further improve the process, and we can stop fooling ourselves that there is such a thing as double blind peer review.
RSS Feed
