A recent editorial in PLOS Medicine discusses the need for transparency in the medical literature, specifically with regard to comparative effectiveness research (CER). The editorial makes many excellent points, but also puts into clear relief the double standard that is consciously being fabricated by proponents of non-science-based medicine.
I wrote previously about another editorial that took a very different approach. Speaking for The Kings Fund, Professor Dame Carol Black said.
“The challenge is to develop methods of research that allow us to assess the value of an approach that seeks to integrate the physical intervention, the personal context in which it is given, and non-specific effects that together comprise a particular therapy.”
The editorial essentially defended the use of CER and other forms of evidence to bolster the evidence base for so-called CAM (complementary and alternative medicine) in order to promote its use.
What I find most interesting is that when medical scientists write in the literature about medical research and practice, absent the context of CAM, their statements and perspective are unapologetically rational and scientific. They typically will advocate for a high standard of science and evidence in making medical policy and decisions. In the context of CAM, however, this standard vanishes and we encounter squirrely apologetics for lowering the scientific standard.
Read the PLOS Medicine editorial and apply the principles they endorse to CAM, and I think you will see what I mean.
With that in mind, let’s turn to the content of the PLOS Medicine editorial, by the journal’s editors. They emphasize the need for publishing negative studies, and lament the “file drawer effect” that is known to bias the published literature toward positive studies. they write:
…the demonstration that an existing practice is ineffective or potentially harmful can (or should) prompt a rapid change in research agendas, policy, and clinical care. Knowing what doesn’t work is particularly useful in efforts to control medical spending, where redirecting limited resources away from ineffective interventions is of obvious benefit.
This is a cornerstone of science-based medicine – actually responding to quality published evidence by altering practice, including stopping practices that are shown not to work or to have unacceptable risks. While mainstream medicine can be criticized for ignoring this rule in some specific cases (while generally applying it), the culture of CAM seems to be based upon ignoring it.
The authors then get to the primary point of their article -that comparative effectiveness research (CER), which has recently received a great deal of funding, requires just as much transparency as traditional efficacy trials in order to ward off distortion of the medical literature.
In short, CER is designed to give real-world comparisons of the applications of various medical interventions (whether diagnostic, preventive, therapeutic, etc.). Such trials are not designed to determine efficacy. I wrote previously about the fact that CER type studies are being abused by some to draw efficacy conclusions – but this is not what such trials are designed to do. For example, they are rarely blinded and do not control for many variables.
But they are useful in assessing the effect on the health care system of various interventions. They just need to be used appropriately.
The PLOS Medicine editors warn that CER has a higher potential for distortion than efficacy trials. They write:
In CER, which will include studies of many types, practices that distort the scientific evidence base—such as ‘‘cherry picking’’ for publication only those studies describing a desired outcome, or ‘‘fishing’’ from an ocean of possible analyses only those that might support favorable (but statistically invalid) conclusions—have the potential to affect policy, practice, and profits to an even greater degree than they have done in the context of traditional efficacy trials.
I agree with their assessment that this means the need for transparency in CER is great. But is also highlights one of the reasons why CER should not be used to determine efficacy, especially of controversial or implausible health claims – and especially as a method to rescue modalities which have failed in efficacy trials.
That is, however, exactly the new approach Dame Carol Black and others who promote CAM have been suggesting.
Conclusion
CER is a useful area of medical research that will help us to navigate toward more cost effective medicine. However, because of the nature of such research, as the PLOS Medicine editors point out, the potential for distortion is great and therefore we need both high standards in research and transparency to minimize known sources of distortion.
For example they specifically recommend (and I agree) extending to CER the requirement to register all clinical trials prior to their onset, so that all results will be made public, not just those favorable to those funding or carrying out the study or those profitable to the study publisher.
They also take the opportunity to endorse open-access journals, which can be seen as a bit self-serving since PLOS is a major open-access medical journal. But their point is legitimate, as open-access journals remove the profit motive from deciding which studies to publish, which creates a motivation to publish positive studies that will garner readers, media attention, and requests for reprints.
In addition, although perhaps not expressly intended by the authors, their points also can be used to argue that CER should not be used to make efficacy claims in the absence of quality efficacy research showing that a specific intervention works.
And finally, the high standards of science and evidence that is overtly being recommended by the PLOS Medicine editors should apply to all of medicine – a single, fair, and thoughtful science-based standard. The call for a double standard or the weakening of the scientific standard of medicine, in order to promote preferred modalities that have not been able to demonstrate efficacy, should be vigorously opposed.
