The overall goal of science-based medicine is to maintain and improve the standard of science in the practice of medicine at every level. At the heart of the scientific basis of medical knowledge and practice is a process known as peer-review. We have occasionally written about peer-review on SBM, and once again the process is under the microscope over a specific question – should peer review be open or blinded?
What is peer review
The term peer-review refers to a pre-publication process in which a journal editor will send a submitted manuscript to 2-3 experts in a field relevant to the paper to carefully examine every aspect of the paper. They then provide a detailed analysis of the paper: is the research question relevant and appropriate, did the study design properly address the question, were the methods rigorous, was the statistical analysis appropriate, was the data presented fairly, are the conclusions supported by the data, did the authors account for other publications addressing the same issue, and did they address every possible question or objection?
The reviewers’ reports will make specific recommendations for changes that would be necessary to improve the paper, and also make their bottom-line recommendation: accept as is, accept with revisions, or deny. The journal editor(s) then rely upon those reviews, plus their own assessment, to make a final decision.
The quality of a scientific journal is directly tied to the quality of peer review, which is a key part of the overall editorial quality.
Open vs blinded peer review
Traditionally most peer review is single-blinded. This means that the reviewers know who the authors are, but the authors do not know who the reviewers are. The reviewers remain anonymous (except to the editors). Authors may suggest reviewers to the editors, but the editors ultimately pick them.
The idea behind blinded reviews is that reviewers will feel free to express their true opinions, without worrying about the political fallout, or making enemies with a colleague if they recommend their paper be rejected. This also avoids tit-for-tat reviews, where a reviewer gives a negative review because one of the authors previously gave one of their papers a bad review, or colleagues exchanging positive reviews.
Some journals are experimenting with an open peer review process, in which the reviewers are publicly known, and their reviews are entirely transparent. In this model the transparency is the mechanism of quality control. The idea is that reviewers will do a thorough and fair job of peer review because all their colleagues will be able to see the job they did. If they were sloppy, or unfairly positive or negative, their peers will see it.
Open peer review should not be confused with open-access peer review. Val Jones wrote about this previously – open access peer review means that anyone can review a paper, which is made available in a public space for anyone to review and comment. This isn’t even really “peer review” – it’s just open access review, because the reviewers don’t have to be peers. This is the crowd-sourcing/Wikipedia approach to peer-review.
As Val pointed out, Peter Frishauf, the founder of Medscape, published a webcast editorial in 2008 predicting that:
Peer review as we know it will disappear. Rather than the secretive prepublication review process followed by most publishers today, including Medscape, most peer review will occur transparently, and after publication.
This seems like a premature prediction. I suspect open-access post-publication review will supplement, and not replace, peer-review. Or, this kind of system may exist parallel to the peer-reviewed published literature.
Head-to-head comparison
There doesn’t seem to be a clear choice between blinded and open peer-review, as both have their strengths and weaknesses. A recent study in BMJ Open directly compared blinded vs open peer review: “Retrospective analysis of the quality of reports by author-suggested and non-author-suggested reviewers in journals operating on open or single-blind peer review models.”
The study compared 200 reviews for articles each from open and blinded journals that were otherwise similar:
BMC Infectious Diseases and BMC Microbiology are similar in size, rejection rates, impact factors and editorial processes, but the former uses open peer review while the latter uses single-blind peer review.
They also did a second comparison. BMC Inflammation was open and then transitioned to blinded peer review, so they compared 200 reviews from the open period with 200 reviews from the blinded period.
In all cases the reviews were independently evaluated and given a standard assessment of quality.
The study also tracked the results of author-suggested reviewers vs other reviewers. What they found was that the quality of reviews submitted by author-suggested reviewers was the same as other reviewers under both models. However, author-suggested reviewers were much more likely to recommend acceptance. They also found that editors put less weight on author-suggested reviewers than other reviewers.
Regarding open vs blinded reviews, the results were mixed. When comparing BMC Infectious Diseases (open) to BMC Microbiology (blinded) they found that the quality of the open reviews on average scored 5% better than the blinded reviews. This result was barely statistically significant at p=0.042.
When comparing BMC Inflammation before and after transitioning from a blinded to an open peer review model, they found no difference in quality.
Conclusion – both methods seem valid
The bottom line of this study is that there does not seem to be much of a difference between the quality of reviews from an open vs blinded peer review model. From the press-release of the study:
Maria Kowalczuk, lead author, BioMed Central’s Biology Editor for the Research Integrity Group and co-Editor-in-Chief of Research Integrity and Peer Review, said: “As advocates of openness, we are excited to find that upon analysis reviewer reports under open peer review are of comparable, or even higher, quality than those of the more established model of single-blind peer review. However, we appreciate our results do not undermine the single-blind model of peer review.
Given that the blinded peer review model is more established, even showing no difference would tend to validate the newer open peer-review model. Of course, this one study does not end the debate, as it was limited to only three journals within one publishing group.
This study also does not address concerns about the downstream effects of open vs blinded review on the culture of science. Do scientists want their reviews to be public, and do they want to know who their reviewers are? How will this affect the community of scientists? These questions are much more difficult to assess.
As the two models seem roughly equivalent (if these results hold up), it does not seem like an objective measure like review quality is going to settle the debate.