Science-based medicine rests on twin pillars that are utterly essential to the development of treatments that are safe and efficacious. Both of these pillars depend on science, but in different ways. The first of these is, of course, the basic science that provides the hypotheses to test about the mechanisms behind the diseases and malfunctions that plague the human body. This basic science suggests ways of either correcting or alleviating these malfunctions in order to alleviate symptoms and prevent morbidity and mortality and how to improve health to increase quality and quantity of life. Another critical aspect of basic science is that it also provides scientists with an estimate of the plausibility of various proposed interventions, treatments and cures designed to treat disease and improve health. For example, if a proposed remedy relies upon ideas that do not jibe with some of the most well-established laws in science, such as homeopathy, the concepts behind which violate multiple laws of physics and chemistry, it’s a very safe bet that that particular treatment will not work and that we should test something else. Of course, the raison d’être of this blog derives from the unfortunate fact that in today’s medicine this is not the case and we are wasting incredible amounts of time, money, and lost opportunities in order to pursue the scientific equivalent of fairy dust as though it represented a promising breakthrough that will save medicine, even though much of it is based on prescientific thinking and mysticism. Examples include homeopathy, reiki, therapeutic touch, acupuncture, and much of traditional Chinese medicine and Ayurveda, all of which have managed to attach themselves to medical academia like kudzu.
Of course, basic science alone is not enough. Humans are incredibly complex organisms, and what we consider to be an adequate understanding of disease won’t always result in an efficacious treatment, no matter how good the science is. Note that this is not the same thing as saying that utter implausibility from a scientific basis (as is the case with homeopathy) doesn’t mean a treatment won’t work. When a proposed treatment relies on claiming “memory” for water that doesn’t exist or postulates the existence of a “life energy” that no scientific instrument can detect and the ability to manipulate that life energy that no scientist can prove, it’s a pretty safe bet that that treatment is a pair of fetid dingo’s kidneys. Outside of these sorts of cases, though, clinical trials and epidemiological studies are the second pillar of science-based medicine, in particular clinical trials, which is where the “rubber hits the road,” so to speak. In clinical trials, we take observations from the laboratory that have led to treatments and test them in humans. The idea is to test for both safety and efficacy and then to begin to figure out which patients are most likely to benefit from the new treatment.
Over the last 50 or 60 years, for all its flaws (and what system devised by humans doesn’t have flaws?) it’s been a highly effective system. When it works well, physicians take observations from the clinic, go to the laboratory, where basic scientists and physicians try to figure out what’s going on to explain a particular observation and then develop an intervention, after which that intervention, be it a drug, procedure, or other treatment, is taken back to the clinic to test. In practice, this process can be very messy, as biases such as publication bias, selection bias, and other confounding factors can at times mislead. Money can corrupt the process as well, given that clinical trials are the final common pathway to the approval of new drugs by the Food and Drug Administration and the hundreds of millions of dollars it costs pharmaceutical companies to bring a single drug from bench to final phase III clinical trials in the hopes of recouping that investment and making large profits besides. Despite all that, no one has as yet been able to propose a better process.
That’s not to say that periodically there aren’t proposals to radically reinvent the clinical trials process. Certainly, I can sympathize to a point; being involved in clinical trials myself, I understand how even a relatively small clinical trial involves an enormous amount of time, money, and regulatory hurdles to jump over. I’ve never personally run a large phase III trial (although I hope to some day); so I can only know what that would be like from my interactions with colleagues who have. In any case, it’s the onerous nature of the current clinical trial system that has led to a recent editorial published in Science by Andrew Grove, former Chief Executive Officer of Intel Corporation and a patient advocate at the University of California, San Francisco, entitled, appropriately enough, Rethinking Clinical Trials. From the article, it’s obvious that Groves is not a scientist, but that doesn’t mean he isn’t worth listening to—to a point. Unfortunately, his proposed solution is unlikely to work, even though he does have a grasp of the problem:
The biomedical industry spends over $50 billion per year on research and development and produces some 20 new drugs. One reason for this disappointing output is the byzantine U.S. clinical trial system that requires large numbers of patients. Half of all trials are delayed, 80 to 90% of them because of a shortage of trial participants. Patient limitations also cause large and unpredicted expenses to pharmaceutical and biotech companies as they are forced to tread water. As the industry moves toward biologics and personalized medicine, this limitation will become even greater. A breakthrough in regulation is needed to create a system that does more with fewer patients.
Groves does have a point in that the clinical trial system in this country has indeed become quite expensive and unwieldy. He’s also correct that the evolution towards “personalized medicine” will exacerbate the problem. The reason is that, as we check more and more biomarkers or genetic markers to guide therapy, we will decrease the number of patients falling into each category requiring a certain drug, in essence, slicing and dicing the patient population into ever smaller slivers, each of whose treatment will be different. Sorting all this out will be quite difficult. Unfortunately, Groves approaches the problem from the wrong perspective in that it’s clear he has little feeling for how science should be applied to medicine, as will become clear by his analogy:
The current clinical trial system in the United States is more than 50 years old. Its architecture was conceived when electronic manipulation of data was limited, slow, and expensive. Since then, network and connectivity costs have declined ten thousand–fold, data storage costs over a million-fold, and computation costs by an even larger factor. Today, complex and powerful applications like electronic commerce are deployed on a large scale. Amazon.com is a good example. A large database of customers and products form the kernel of its operation. A customer’s characteristics (like buying history and preferences) are observed and stored. Customers can be grouped and the buying behavior of any individual or group can be compared with corresponding behavior of others. Amazon can also track how a group or an individual responds to an outside action (such as advertising).
Yes, you heard that right. Groves thinks that doing science is enough like cataloging customer orders, preferences, and history the way Amazon.com does. So what’s his suggestion? In essence, Groves is proposing what is commonly known a “pragmatic trial” but on megadoses of steroids, all using computers to figure out what’s going on:
We might conceptualize an “e-trial” system along similar lines. Drug safety would continue to be ensured by the U.S. Food and Drug Administration. While safety-focused Phase I trials would continue under their jurisdiction, establishing efficacy would no longer be under their purview. Once safety is proven, patients could access the medicine in question through qualified physicians. Patients’ responses to a drug would be stored in a database, along with their medical histories. Patient identity would be protected by biometric identifiers, and the database would be open to qualified medical researchers as a “commons.” The response of any patient or group of patients to a drug or treatment would be tracked and compared to those of others in the database who were treated in a different manner or not at all. These comparisons would provide insights into the factors that determine real-life efficacy: how individuals or subgroups respond to the drug. This would liberate drugs from the tyranny of the averages that characterize trial information today. The technology would facilitate such comparisons at incredible speeds and could quickly highlight negative results. As the patient population in the database grows and time passes, analysis of the data would also provide the information needed to conduct postmarketing studies and comparative effectiveness research.
I found out about Andy Groves’ article from Derek Lowe, who didn’t think that much of it but didn’t dismiss it altogether. I tend to agree, although I suspect I’ll end up being a little bit harder on it than Derek is. And it’s not an altogether crazy idea. It’s not even necessarily that bad an idea, except that Groves clearly doesn’t understand clinical trials, and you have to understand clinical trials before you can apply technology to it. For example, Groves seems to labor under the delusion that phase I trials prove safety of a new medication. That is a gross misunderstanding of the purpose of the phase I trial. Yes, checking for safety is part of what a phase I trial does, but a phase I trial doesn’t “prove safety.” What a phase I trial does is to rule out any really major side effects or toxicities that are common (remember, phase I trials usually only have around 20 to 100 participants, too small a number to catch uncommon adverse events), study pharmacokinetics (how the drug level varies with dose and how it’s metabolized), and establish both a maximal tolerated dose and a dosing interval. This last purpose is usually achieved using a technique as dose escalation Often phase I trials are performed using healthy volunteers, although in my specialty (cancer) that’s rarely the case. In any case, a better way of describing the purpose of a phase I was summed up by Freedman, “[T]he reason for conducting the trial is to discover the point at which a compound is too poisonous to administer.” That’s exactly what I meant by “maximal tolerated dose.”
Yes, that is the purpose of a phase I “first in humans” clinical trial. It’s absolutely necessary, too.
Here’s the problem with Groves’ idea. What he is basically proposing is to do, in essence, a whole bunch of “N of 1” trials, each patient being a clinical trial in and of himself or herself. Then, through the magic of computer technology, he seems to be suggesting that we take all these “N of 1” trials and try to do a meta-analysis of them. Here, we have a case where more does not necessarily mean better. What will result are data that are ridiculously heterogeneous—possibly unanalyzably so. As Derek Lowe points out, one of the most difficult aspects of clinical trial design is to standardize the treatment, to make sure that patients across multiple clinical trial sites are actually being treated and followed in the same way. Under Groves’ concept, heterogeneity is a feature, not a bug. However, it is not this aspect that bothers me so much about this proposal. Rather, it’s Groves’ dismissive comment about “liberating” clinical trials from the “tyranny of averages.” As if averages are a bad thing! That “tyranny of averages” is what makes sure that the patients being enrolled in a clinical trial are comparable to each other. Without relatively strict inclusion criteria in early phase II trials, the most likely thing that would happen if Groves’ proposal were adopted is that any signal would be drowned out by all the noise due to the heterogeneity of the patients and the data derived from each “N of 1” trial.
Perhaps the biggest practical problem with Groves’ idea is how patients will be selected for therapies. Notice how Groves says that “once safety is proven, patients could access the medicine in question through qualified physicians.” There’s another problem with this concept other than the lack of recognition of the fact that phase I trials don’t “prove safety,” and that’s the issue of who decides which patients will take the drug, and basically it appears to me that what Groves is proposing is that any physician can take any drug that has passed phase I testing and offer it to any patient. As much as Groves prattles on about “real world” efficacy, this is a real world recipe for disaster. First, phase I trials do not demonstrate efficacy; they only evaluate safety and toxicity. Consequently, it is difficult (for me, at least) to imagine how physicians could ethically administer drugs whose efficacy has not been demonstrated or, more importantly, how they could know for which patients these new drugs would be appropriate. (Short answer: They can’t.”) It’s difficult enough to maintain clinical equipoise.
Indeed, one huge unspokean (and unsupported) assumption is that allowing unfettered access to experimental drugs that have passed phase I trials would help more people than it would hurt. In actuality, because phase I trials only identify acute toxicities and do not identify adverse reactions that occur with longer use, physicians administering these drugs would be flying almost blind. The potential for harm is enormous, particularly when it is powerful chemotherapeutic agents that are being given. It is far more likely that widespread use of unapproved substances would harm far more patients than it would help. Indeed, at the level of the individual patient, trying such drugs is more likely to harm than help. If there’s one thing worse than dying of cancer, it’s making one’s last days shorter and more miserable with toxicities from unapproved drugs or, even worse still, paying big bucks to do so.
Yet, somehow Groves seems not to have considered this possibility.
Perhaps the most problematic aspect of Groves’ entire proposal, though, is the very reason why we do clinical trials, namely to answer the question, “Does the drug work?” In a system such as that proposed by Groves, how, exactly, would we figure out whether a drug works or not? What would be the endpoint? What result would tell us that the drug is doing what it is intended to do? For example, in the case of cancer chemotherapy drugs, the purpose of the drug is to prolong survival. Figuring out if a new drug did that is difficult enough in the current system of clinical trials. Indeed, we already know from the example of Avastin in breast cancer that teasing out whether an improvement in disease-free survival translates into an improvement in overall survival. Under Groves’ proposal, it would be well nigh impossible. Groves seems to be arguing that, if we just keep track of enough variables and possible confounding factors, everything will shake out due to the wonder of modern computerized “e-commerce”-style tracking applied to patients. Maybe that’s possible. Maybe (and more likely) such a system will result in an uninterpretable mass of data from which extracting meaningful correlations will be at best problematic, at worst impossible. Even if it does work, then what is the endpoint of a clinical trial? When can investigators declare that they’ve accrued enough patients.
Remember how I referred to Groves’ proposal as being in essence replacing the current clinical trial system with that of “pragmatic trials”? We’ve been very critical here at SBM of the use and abuse of pragmatic trials by proponents of quackademic medicine. In fact, more than anything else, what Groves is proposing comes across to me as a high tech version of the very same pragmatic trials that acupuncturists are agitating for. There are no controls, which means that placebo responses will go uncorrected for. There are a plethora of variables and potential confounding factors, which would also be unaccounted for.
Don’t get me wrong. I’m not dismissing Groves’ idea; I’m merely pointing out that he has an incredibly simplistic view of how clinical trials operate and what evidence must be obtained before it’s reasonable to conclude that a new treatment “works” for a particular illness. Basically, spurred on by his own personal battles with prostate cancer and Parkinson’s disease, he has had a late life conversion to patient advocate. There’s nothing wrong with that and much to be admired, but unfortunately Groves seems to think that his knowledge of the computer and semiconductor industry is easily transferable to the pharmaceutical industry. It’s not for nothing that four years ago Derek Lowe also referred to Groves as Rich, Famous, Smart, and Wrong. Groves expresses frustration at the slow pace of research into Parkinson’s disease and other diseases. Fair enough. If I had a relentless degenerative neurological condition that would slowly rob me of my ability to function (and, in particular, to do surgery and write), I’d be frustrated too. Unfortunately, he doesn’t seem to understand that medicine is not the semiconductor industry. There’s a reason why we haven’t cured cancer yet, for example. It’s damned hard, and biomedical research does not lend itself easily to the sort of deadline-driven mentality that Groves had as CEO of Intel.
Derek Lowe put it well:
Mr. Grove, here’s the short form: medical research is different than semiconductor research. It’s harder. Ever seen one of those huge blow-ups of a chip’s architecture? It’s awe-inspiring, the amount of detail that’s crammed into such a small space. And guess what — it’s nothing, it’s the instructions on the back of a shampoo bottle compared to the complexity of a living system.
That’s partly because we didn’t build them. Making the things from the ground up is a real advantage when it comes to understanding them, but we started studying life after it had a few billion years head start. What’s more, Intel chips are (presumably) actively designed to be comprehensible and efficient, whereas living systems — sorry, Intelligent Design people — have been glued together by relentless random tinkering. Mr. Grove, you can print out the technical specs for your chips. We don’t have them for cells.
And believe me, there are a lot more different types of cells than there are chips. Think of the untold number of different bacteria, all mutating and evolving while you look at them. Move on to all the so-called simple organisms, your roundworms and fruit flies, which have occupied generations of scientists and still not given up their biggest and most important mysteries. Keep on until you hit the lower mammals, the rats and mice that we run our efficacy and tox models in. Notice how many different kinds there are, and reflect on how much we really know about how they differ from each other and from us. Now you’re ready for human patients, in all their huge, insane variety. Genetically we’re a mighty hodgepodge, and when you add environment to that it’s a wonder that any drug works at all.
It is, indeed.
None of this is to imply that we can’t improve our clinical trials system. As has been pointed out, it’s hugely expensive and inefficient, and these problems are getting worse with the evolution of drug treatment towards “personalized medicine.” We are going to have to figure out ways to make clinical trials smaller and more targeted. We are also going to have to find ways to extract every last bit of information and benefit out of every last clinical trials subject. An approach such as what Groves proposes might well contribute to achieving that aim, particularly when coupled with new trial designs that emphasize the incorporation of biomarkers for drug response. Contrary to Andy Groves’ claims, however, there is no way his sort of approach will ever replace well-designed clinical trials.