Peer-review is a critical part of the functioning of the scientific community, of quality control, and the self corrective nature of science. But it is no panacea. It is helpful to understand what it is, and what it isn’t, its uses and abuses.
Overview
When the statement is made that research is “peer-reviewed” this is usually meant to refer to the fact that it has been published in a peer-reviewed journal. Different scientific disciplines have different mechanisms for determining which journals are legitimately peer-reviewed. In medicine the National Library of Medicine (NLM) has rules for peer-review and they decide on a case by case basis which journals get their stamp of approval. Such journals are then listed as peer-reviewed.
The basic criterion is that there is a formalized process of peer-review prior to publication – so this presents a barrier to publication that acts as a quality control filter. Typically, the journal editor will give a submitted paper to a small number of qualified peers – recognized experts in the relevant field. The reviewers will then submit detailed criticism of the paper along with a recommendation to reject, accept with major revisions, accept with minor revisions, or accept as is. It is rare to get an acceptance as is on the first round.
The editor also reviews the paper, and may break a tie among the reviewers or add their own comments. The process, although at times painful, is quite useful in not only checking the quality of submitted work, but improving the quality. A reviewer, for example, may point out prior research the authors did not comment on, or may point our errors in the paper which can be fixed.
It is typical for authors to submit a paper to a prestigious journal first, and then if they get rejected to work their way down the food chain until they find a journal that will accept it. This does not always mean that the paper was of poorer quality – the most prestigious journals have tons of submissions and can pick and choose the most relevant or important studies. But sometimes it does mean the paper is mediocre or even poor.
The limitations of Peer-Review
It is important to realize that not all peer-reviewed journals are created equal. Small or obscure journals may follow the rules and gain recognized peer-reviewed status, but be desperate for submissions and have a low bar for acceptance. They also have a harder time getting world-class experts to review their submissions, and have to find reviewers that are also farther down the food chain. The bottom line is that when a study is touted as “peer-reviewed” you have to consider where it was reviewed and published.
Even at the best journals, the process is only as good as the editors and reviewers, who are people who make mistakes. A busy reviewer may give a cursory read through a paper that superficially looks good, but miss subtle mistakes. Or they may not take the time to chase down every reference, or check all the statistics. The process generally works, and is certainly better than having no quality control filter, but it is also no guarantee of correctness, or even the avoidance of mistakes.
Peer-reviewers also have biases. They may be prejudiced against studies that contradict their own research or their preferred beliefs. They may therefore bias the published studies in their favored direction, and may be loath to give a pass to a submission that would directly contradict something they have published. For this reasons editors often allow authors to request or recommend reviewers, or to request that certain people not be asked to be reviewers. Each journal has their own policy. Sometimes an editor will specifically use a reviewer that the authors request not be used, thinking they may be trying to avoid legitimate criticism.
The process can be quite messy, and full of politics. But in the end it more or less works. If an author thinks they were treated unfairly by one journal, they can always go to another or they can talk directly to the editor to appeal a decision and try to make their case.
Perhaps the biggest weakness of peer-review, however, is when an entire discipline of peers is lacking in some fundamental way. For example, there are now many journals dedicated to so-called “alternative medicine” (CAM). The editors of such journals tend to have a pro-CAM bias, and they find reviewers with a pro-CAM bias. So pretty much any pro-CAM article can get published. Some have enough ideological friends at the NLM that they can get approved as peer-reviewed, despite glaring biases in their editorial policy.
Post Publication Peer Review
The term peer-review is sometimes used to refer to the fact that papers are read and reviewed by the broader scientific community once they are published. However, this post-production review should not be confused with “peer-reviewed” and that term should not be used to refer to post-publication review, to avoid confusion.
The process, however, is even more critical to quality control in science. Now, instead of one editor and 2-3 reviewers looking at a study, dozens or hundreds (maybe even thousands) of scientists can pick over a study, dissect the statistics and the claims, bring to bear knowledge from related areas or other research, and provide detailed criticism. This is the real “meat grinder” of science. Hundreds of reviewers are more likely to find problems than the few pre-publication reviewers. Arguments can be tested in the unforgiving arena of the scientific community, weeding out bad arguments, honing others, so that only the best survive.
Conclusion
Here is the bottom line – peer-review is a necessary component of quality control in science, but is no guarantee of quality, and you have to know the details of the journal that is providing the peer-review.
