Peer-review has been the cornerstone of quality control in academia, including science and medicine, for the past century. The process is slow and laborious, but a necessary filter in order to maintain a certain standard within the literature. Yet more and more scholars are recognizing the speed, immediacy, and openness of the internet as a tool for exchanging ideas and information, and this is causing some to question the methods of peer review. A recent New York Times article discusses this issue.
This issue is very relevant to Science-Based Medicine as this is in part an experiment – an attempt to produce a high quality, editorially filtered, but not peer-reviewed, online journal. Our process here is simple. Outside submissions are reviewed by two or more editors and typically are either accepted with minor revisions or rejected. In addition we have a staff of regular contributors – those who have a proven track record of producing high quality articles. There is no pre-publication review for their submissions, and they are able to post directly to SBM.
Because many of the issues we cover are timely, we emphasize speed of publication. Therefore copy-editing is done post-publication – the notion being that our readers can tolerate a few typos in order to gain access to material more quickly.
In addition our pieces are, essentially, crowd sourced. SBM editors and readers provide feedback in the comments, often pointing out ambiguous wording or even outright errors, which can then be quickly fixed or clarified. Editors also provide direct feedback to authors if a more serious issue emerges (which is rare, but happens) and steps are taken to transparently fix such issues. This is, in a way, post-publication peer-review.
By contrast, traditional peer-review is designed to take a long time to produce the highest quality article prior to going to print (even if “print” is online). The strength of this process is that several editors and peer-reviewers have thoroughly gone over the submission, corrected errors, fixed ambiguities, added missing insights, reviewed methodology, checked references, and made sure that the author’s conclusions do not overstep the data. At least this is the ideal – we often write about peer-reviewed publications on SBM that fall far short of this ideal. In the end, peer-review is only as good as the editors and reviewers. I have both been a writer and reviewer, and the process does have many strengths. In the end the article that gets printed is much improved over the original submission.
The weaknesses of the process, however, are the long delay to publications – months or even years. Further, the process is a bit hit or miss and depends largely on the quality of the reviewers. Individuals are quirky, and they may have biases or missing information that will hamper the quality of their review. The more people there are involved in the process the better the quality is likely to be.
As academia grapples with the internet age, the challenge is to rethink the process of peer-review. Specifically, are there ways to leverage the power of the internet to make the review process faster and better? This has happened in the technical sense – the traditional process of peer review is now often done online. For the last article I reviewed I was not sent a hard copy or even an electronic copy. I signed onto a secure website where I had access to the article, and I submitted my review entirely online. This was convenient and also a time-saver. But still the process was slowed by the need to choose reviewers, wait for them to accept and then for those specific reviewers to review the paper. And in the end opinions were solicited from only two reviewers.
Imagine an alternate process by which an article is published online, either on an open site or a secure site that only experts have access to. Then dozens or hundreds of experts can comment on the paper, providing feedback directly to the authors in addition to the editors, who can also respond to the commenters. The result would be more of a dynamic conversation than you get with the current review process. But most importantly, in my opinion, is that you would get a broader range of opinions, and a far greater chance to detect error or bias. An editor or editors can oversee the process, and once it has played itself out the final version of the paper can be published to the public, and become part of the official literature.
Now is the time for experimentation, in my opinion. I don’t know yet what the optimal system would be, but I think a hybrid of peer-review and open publication will emerge. There still needs to be some filter, and some editorial process selecting articles, overseeing feedback and revisions, and approving the final version. Feedback from many experts should be solicited in some fashion, and the process should be interactive.
There are many details to experiment with, however. Should commenters be named or anonymous? Anonymous feedback may be more candid, but people should be held accountable for their expert opinions as well. How should commenters be limited, if at all? Should there be a period of feedback open to the public, or should pre-publication access only be granted to experts, and how should that be limited? (How narrow a field of experts should apply?)
There is also the issue of presenting new research in an open forum prior to adequate review. Will the press troll these pre-review sites looking for juicy stories based upon flawed research that would never ordinarily make it to publication?
The strengths and weaknesses of any new process of peer-review will not be fully appreciated until it is implemented, which is why we are in a period of experimentation. But the experiment is worth doing. In the end we may gain a better, faster and more transparent review process.
