One of our heroes at SBM: John Ioannidis.
It’s no secret that I’m a fan of John Ioannidis. So, I daresay, are pretty much all of the editors and regular contributors to this blog. (If you don’t believe me, just type Ioannidis’ name into the blog search box and see how many posts you find.) Over the last couple of decades, Ioannidis has arguably done more to reveal the shortcomings of the medical research enterprise that undergirds our treatments, revealing the weaknesses in the evidence base and how easily clinical trials can mislead, than any other researcher. Indeed, after reading what is Ioannidis’ most famous article, “Why Most Published Research Findings Are False“, back in 2005, I was hooked. I even used it for our surgical oncology journal club at the cancer center where I was faculty back then. This was long before I appreciated the difference between science-based medicine (SBM) and evidence-based medicine (EBM). So it was with much interest that I read an article by him published last week and framed as an open letter to David Sackett, the father of evidence-based medicine, entitled “Evidence-based medicine has been hijacked: a report to David Sackett.” Ioannidis is also quoted in a follow-up interview with Retraction Watch.
Before I get to Ioannidis’ latest, I can’t help but point out that, not surprisingly, quacks and proponents of pseudoscientific and unscientific medicine often latch on to Ioannidis’ work to support their quackery and pseudoscience. They’ve been doing it for years. Certainly, they’re already latching on to this article as vindication of their beliefs. After all, their reasoning—if you can call it that—seems to boil down to: If “conventional” medicine is built on such shaky science, then their pseudoscience isn’t wrong after all, given that the same scientific enterprise upon which conventional medicine is based produces the findings that reject their dubious claims and treatments. Of course, whenever I hear this line of argument, I’m reminded of Ben Goldacre’s famous adage, seen in one form on Twitter here:
Quacks citing problems in pharma make me laugh. FLAWS IN AIRCRAFT DESIGN DO NOT PROVE THE EXISTENCE OF MAGIC CARPETS.
— ben goldacre (@bengoldacre) January 31, 2013
The adage can be generalized to all EBM and SBM as well. Just because big pharma misbehaves, EBM has flaws, and conventional medicine practitioners don’t always use the most rigorous evidence does not mean that, for example, homeopathy, acupuncture, or energy medicine works.
Still, when Ioannidis publishes an article with a title provocatively declaring that EBM has been “hijacked,” we at SBM take notice.
Has EBM been “hijacked”?
In his “report” to David Sackett, Ioannidis does touch on a number of pertinent and interesting points regarding the adoption of EBM but, as you will see, pretty much ignores the one huge elephant in the room. Before you hear the pounding of elephant feet, though, let me just say that I think Ioannidis is being a bit too hard on himself to have declared himself a “failure” in 2004 and an “even bigger failure” twelve years later. I suppose by the metrics he uses he is a failure, but I would argue that those are the wrong metrics. First, however, let’s see where Ioannidis is coming from. He believes that EBM met considerable resistance in the US:
EBM met with substantial resistance in the 1990s and 2000s. Even in the USA, the mecca of biomedical research, EBM, and any serious biomedical research that may help intact humans was largely unwanted. As a clinical research fellow, I remember that every week we were waiting to hear whether the Agency for Health Care Policy and Research (AHCPR, which subsequently became AHRQ) would be axed. The agency had hurt the interests of some powerful professional surgical society: one of its guidelines threatened the indications for an expensive and possibly largely useless surgical procedure. AHCPR/AHRQ survived, but has always had to fight valiantly for its existence since then. EBM is widely tolerated mostly when it can produce largely boring evidence reports that are shaped and endorsed by experts. Many years later, the Patient-Centered Outcomes Research Institute was launched, an equally valiant effort to cover some of that vast space. It was also restricted by a mandate not to deal with cost effectiveness of interventions. I was asked to participate in its Methodology Committee along with great colleagues. I contributed far less than anyone else to the effort before deciding to quit out of shame for my lack of contribution. Most of our tasks seemed to require experts rather than evidence. More than 7 billion of people would be better qualified than me to lead expert-based activities.
One can’t help but wonder to which surgical procedure Ioannidis is referring to. Could it be vertebroplasty? Or is Ioannidis referring to the 1990 report by the AHRQ that found rest and pain medications worked as well as surgery for back pain? I think that’s probably it. Be that as it may, it is true that political forces in medicine have long detested the AHRQ. Critics argue that its research is duplicative with the NIH, even though the NIH does not generally fund studies of the health care delivery system as the AHRQ does, and PCORI, even though the PCORI funds only comparative effectiveness research. (Comparative effectiveness research compares the effectiveness of already approved and utilized medical interventions.) Moreover, the agencies all work very hard to avoid duplication.
On the other hand, in organized medicine there is a large constituency that supports the AHRQ, which was in evidence last year during the latest attempt to defund the organization, when dozens of health care organizations raced to the agency’s defense and more than 100 health groups signed a letter in June and July urging the House and Senate to maintain AHRQ and the research it funds, including the American College of Surgeons and the American Association of Orthopedic Surgeons. I also can’t help but note the development of Choosing Wisely four years ago, where physician professional societies consciously agreed to start making lists of interventions that remain common but are not evidence-based in an effort to decrease their use.
So, yes, there is resistance to the AHRQ. However, these days it is far more business interests, such as drug and device manufacturers, than physicians groups who want to abolish both the AHRQ and the PCORI, mainly because the AHRQ and PCORI’s research threatens these companies’ bottom lines by showing which treatments work better in the “real world” and influencing the Centers for Medicare & Medicaid Services (CMS) regarding which new drugs and devices will be paid for. In fact, I’d argue that, while Ioannidis is correct that drug and device manufacturers want to kill AHRQ and PCORI, he’s missed a sea change in attitude among physicians towards such government agencies whose purpose is to evaluate and compare treatments for effectiveness after they’ve been approved. It might have been true that EBM was not popular 15 or 20 years ago, but as new generations of medical students have been inculcated with its principles and importance, EBM has been “baked in” to physician education, with a resultant change in attitude towards efforts to promote EBM. That’s not to say that physician groups don’t protect their turf. Just look at how radiologists, for example, react to new guidelines that increase the recommended age at which to start mammography or how primary care physicians react to legislation expanding the scope of practice of advanced practice nurses. However, extreme hostility to comparative effectiveness research and EBM-based guidelines has mostly retreated to fringe physician groups like the American Association of Physicians and Surgeons. Unfortunately, resistance to EBM as a constraint on physician autonomy is still fairly common, particularly among older physicians.
I view this as one victory of John Ioannidis and others like him. In other words, I agree with much of what he says, but I tend to have a different interpretation, although not always. For instance, it’s very clear that the attacks on the AHRQ and PCORI have little or nothing to do with EBM being “co-opted,” but rather powerful business interests that see EBM-based guidelines promulgated by AHRQ as inimical to their financial interests. Ioannidis even relates a story of a senior professor of cardiologist telling a friend of his that Ioannidis should not be too outspoken, “otherwise Albanian hit men may strangle him in his office.” To me, that’s evidence of success.
Methodolatry and the rumbling of elephants
Here’s where the elephant in the room that I mentioned earlier starts getting closer. Specifically, what Ioannidis complains about next are definitely signs of EBM’s success, but I also agree that they are signs of EBM being co-opted or “hijacked.” After all, if EBM weren’t successful, then why would anyone want to hijack it? The problem is that somehow Ioannidis, as brilliant as he is, doesn’t seem to realize one part of the solution to the problems of EBM that he’s been relating and relates here.
In any case, here’s the next part of Ioannidis’ tale:
Now that EBM and its major tools, randomized trials and meta-analyses, have become highly respected, the EBM movement has been hijacked. Even its proponents suspect that something is wrong [[10], [11]]. The industry runs a large share of the most influential randomized trials. They do them very well, they score better on “quality” checklists [12], and they are more prompt than nonindustry trials to post or publish results [13]. It is just that they often ask the wrong questions with the wrong short-term surrogate outcomes, the wrong analyses, the wrong criteria for success (e.g., large margins for noninferiority), and the wrong inferences [[14], [15], [16]], but who cares about these minor glitches? The industry is also sponsoring a large number of meta-analyses currently [17]. Again, they get their desirable conclusions [18]. In 1999 at the closing session of the Cochrane Colloquium in Rome, among the prevailing enthusiasm of this benevolent community, I spoiled the mirth with my skepticism. I worried that the Cochrane Collaboration may cause harm by giving credibility to biased studies of vested interests through otherwise respected systematic reviews. My good friend, Iain Chalmers, countered that we should not worry—plus there were many topics where the industry had not been involved. He mentioned steroids as one example. It was not very reassuring. Now even the logo of the Collaboration, the forest plot for prenatal steroids, has been shown to be partially wrong due to partial reporting [19]—let alone reviews of trials done with vested interests from their very conception.
It is, of course, not surprising that industry would run the lion’s share of randomized clinical trials (RCTs). The FDA requires RCTs demonstrating efficacy and safety before it will approve a drug or device, and the drug and device industries are in the business of marketing new drugs and devices. It is thus in their financial interest to run the most “rigorous” trials (from an EBM standpoint) that they can and publish them as quickly as they can, because that will speed up FDA approval. Unfortunately, it is also in their interest to ask questions and use endpoints that are most likely to produce “positive” trials. What Ioannidis neglects to mention is that the pressure to do this also comes from non-financial motivations. You have only to look at the activism of AIDS victims in the 1980s and the rise of the “right to try” movement to realize that there is a large and strong constituency demanding faster drug approval. This constituency has influence, too. As a result, legislators are willing introduce legislation that would weaken scientific standards for drug and device approval, like the Saatchi bill in the UK and the 21st Century Cures Act in the US. Not coincidentally, these bills align with industry interests as well as patients looking for faster approval of new drugs. Of course, the answer to this problem is not necessarily a problem with EBM itself. Rather, it is the regulations that drug approval agencies in the US and other countries use to approve drugs and devices.
As for the Cochrane Collaborative, we’ve discussed problems with this group right from the very beginning, most recently when Tom Jefferson of the Cochrane Acute Respiratory Infections Group argued that childhood vaccination schedules are not evidence-based because, in essence, there are not RCTs testing every schedule and RCT evidence refuting various claimed adverse reactions doesn’t exist. This is a problem with EBM that I, in particular, have discussed multiple times before on this blog, the problem of methodolatry, defined as the “profane worship of the randomized controlled clinical trial as the only valid means of investigation.” While it is true that RCTs are the “gold standard” in clinical evidence of efficacy and safety, there are many questions that for either ethical or practical reasons can’t be studied by RCTs. Unfortunately, methodolatry is a feature, not a bug, of EBM that consigns all evidence below that of RCTs and meta-analyses of RCTs to a category of inferior evidence or not evidence at all! That includes the epidemiological studies that have shown vaccines to be safe and not to be a cause of autism, as well as the entire corpus of basic science that tells us homeopathy is impossible pseudoscience. In EBM, it’s RCT evidence and, in particular meta-analyses über alles!
This is why we at SBM frequently refer to the “blind spot” of EBM. From our perspective, EBM has indeed been “hijacked.” I, for example, agree with Ioannidis that industry has to some extent hijacked EBM, but we also add that advocates of quackery have also done so, hence the creation of a whole new field of medicine dedicated to “integrating” pseudoscience with medicine, an NIH center (the National Center for Complementary and Integrative Health), and many programs in academic medical centers dedicated to what I like to call quackademic medicine.
What’s frustrating is that Ioannidis seems at some level to recognize this problem but doesn’t explicitly address it. For instance, he mentions in his article that there are “so many quacks ranging from television presenters and movie stars turned into health trainers [40] and pure science denialists (e.g., climate, HIV, vaccine denialists, and religious fundamentalists) that one has to tread carefully” and that we should “avoid a civil war on how to interpret evidence within the health sciences when so many pseudoscientists and dogmatists are trying to exploit individuals and populations and attack science,” but seems unwilling to address the elephant in the room, how EBM has been hijacked by CAM promoters as well as industry.
Science-Based Medicine: At least part of the cure
We at SBM really like John Ioannidis because we, like he, are dedicated to improving the scientific rigor of medicine. What we have a problem with in EBM is the methodolatry that has come to dominate the core of EBM that leads EBM purists to refuse to dismiss obvious quackery like homeopathy and “energy medicine.” In other words, EBM does not adequately take into account prior probability, and as a result demands RCTs for just about everything. I’m not saying that EBM ignores basic science considerations and the prior probability derived from them; it’s just that EBM relegates basic science to the very lowest rung of the EBM hierarchy. So, on the one hand, EBM (mostly) dismisses basic science considerations, and its methodolatry leads to unethical trials of homeopathy and cancer quackery like the Gonzalez protocol, as well as the widespread acceptance of “integrative medicine” at universities and massive research programs studying quackery (a.k.a. quackademic medicine).
In fact, you can see some of this attitude in Ioannidis’ quoting of Sackett:
We value the study of adenine, thymine, cytosine, and guanine far above the study of childhood diarrhoea, Chagas’ disease, community health, and patients’ decision making. The issue is not the potential usefulness of basic research at some distant future date (although you might even dare question whether it is being oversold). The issue is that basic medical scientists have hijacked the granting bodies and have erected research policies that place greater value in serving their own personal curiosities than in serving sick people.
Perhaps this was true in 2004, although I was around then and am still not sure I buy Sackett’s take even then. It’s certainly not true now, where getting a basic science grant funded by the NIH is damned near impossible without a clear disease or translational research angle. There’s a reason why scientists are calling for more basic science. Whether he realizes it or not, by quoting Sackett, Ioannidis inadvertently appears to be echoing the discounting of basic science at the heart of EBM.
Of course, Ioannidis is not wrong when he observes:
With clinical evidence becoming an industry advertisement tool and with much “basic” science becoming an annex to Las Vegas casinos, how about the other pieces of EBM, for example, diagnosis and prognosis and individualizing care? I have had great excitement about the prospects of omics, big data, personalized medicine, precision medicine, and all. Much of my effort has been to put together these efforts with rigorous statistical methods and EBM tools. But I am tired of seeing the same overrated promises recast again and again. For example, several years ago I gave an invited lecture at a leading institution on the danger of making inflated promises in personalized medicine. Right after my talk, everybody rushed to hear the launch of a new campaign, where the leader of the institution singled out this unique historic moment: that institution would single-handedly eliminate most major types of cancer within a few years. Several years have passed, and none of these cancer types have disappeared. I recently tried to find the name of that campaign online but realized that this institution has launched many similar campaigns. Which among many was the unique historic moment that I happened to be at? Multiply this by thousands of institutions, and there are already millions of unique historic moments where cancer was eliminated. Same applies to neurologic diseases and more. I do not understand why academic leaders and politicians need to make such self-embarrassing announcements now and then.
I do. It’s the same reason why industry has tried to co-opt EBM, and Ioannidis is quite correct to condemn this sort of behavior. Over the years we’ve written many posts criticizing the same sorts of claims. I myself have questioned the hype over “precision medicine,” “liquid biopsies,” and the like. I remember a director of the National Cancer Institute making the claim that we would eliminate the suffering and death from cancer by the year 2015. Well, it’s 2016, and cancer is still with us. I still see a lot of suffering and death due to it. I can still remember attending a surgery meeting and shaking my head as I watched a talk touting this initiative.
So Ioannidis has described much of the disease of EBM, and we’ve tried to describe the rest. What’s the cure?
Our fearless leader Steve Novella argues that SBM is the answer. I agree, but not entirely. To me, SBM is a large part of the answer. Much of the problem with EBM that allows it to be hijacked is EBM’s blind spot with respect to basic science. Steve is correct that SBM is designed to take a “big picture”, global look at any clinical claim and that we need to consider the scientific plausibility of the claim, not just RCTs. However, there is a limit to what SBM can do in correcting the co-optation of EBM.
To understand why, just consider: How do we estimate the basic scientific plausibility of a medical claim? It’s not nearly as straightforward as one might think. How plausible a claim is worth investigating with an RCT? 10% likely to be true? 50% likely to be true? 75% likely to be true? What SBM is great for is rejecting ridiculously implausible health claims, like those made by homeopaths, on the basis of basic science considerations alone that tell us that they are impossible. It’s also great for increasing the rigor of clinical trials by taking scientific mechanism into consideration. It can also be useful for prioritizing which clinical trials to do. I would also argue that SBM can counter the overselling of basic science and early clinical trial results that Ioannidis complains about. Finally, I agree with Ioannidis about how many risk factors for disease have been promoted, based on rather shaky evidence. It’s not for no reason that I thoroughly enjoyed his way of illustrating this point by going through a cookbook to look for which food ingredients have been linked with either increasing or decreasing the list of cancer and finding that most of them have, although the evidence was very shaky.
It has to be noted, however, that, as Ioannidis himself noted, industry clinical trials are often among the most rigorous out there, “checking off all the EBM boxes” for rigor. The same observation also applies to the basic science foundations of these trials. Before a drug company even does its first clinical trial of a new drug, it has reams of basic science data. The mechanism is known and documented. It has to be that way, because the FDA requires it. SBM alone is unlikely to have much of an impact on the hijacking of EBM by industry. Sure, it would help to decrease the number of trials that ask the wrong questions or use the wrong analyses. Similarly, applying prior probability to clinical trial analysis would likely decrease the number of seemingly “positive trials” whose results really aren’t clinically significant. However, I suspect that SBM would be less useful with respect to surrogate outcomes, because choosing surrogate outcomes is a question EBM is already well-equipped to deal with, and, aside from clinical trials of “complementary and alternative medicine” treatments with very low prior probability, the same can be said with regard to the criteria for success and how to derive inferences.
What’s far more likely to have an effect are two things: Transparency, and the requirement of more clinically relevant endpoints by both journal editors and government regulatory agencies. Of the two, transparency is likely to be very important, which is why the AllTrials movement is so important, to make sure that the results of every RCT is recorded somewhere, including the negative ones that all too often are not published. As hard as making sure the results of all clinical trials are reported in a timely fashion is, herding the cats to get them to agree on what constitutes a more “clinically relevant endpoint” and a sufficiently rigorous trial design will be more difficult, given the number of journals that publish clinical trial results and the number of government agencies that oversee drug and device approval in various countries.
That’s where the work of people like John Ioannidis will be most important. The first step in solving a problem is to identify it, and no one has been able to identify the problems with how science is applied to medicine as effectively and prolifically as John Ioannidis. I only wish he were more cognizant of the blind spot of EBM with respect to basic science than he appears to be.