While attending a lecture by a naturopath at my institution I had the opportunity to ask the following question: given the extreme scientific implausibility of homeopathy, and the overall negative clinical evidence, why do you continue to prescribe homeopathic remedies? The answer, as much as my question, exposed a core difference between scientific and sectarian health care providers. She said, “Because I have seen it work in my practice.”
There it is. She and many other practitioners of dubious modalities are compelled by anecdotal experience while I am not.
An anecdote is a story – in the context of medicine it often relates to an individual’s experience with their disease or symptoms and their efforts to treat it. People generally find anecdotes highly compelling, while scientists are deeply suspicious of anecdotes. We are fond of saying that the plural of anecdote is anecdotes, not data. Why is this?
Humans are social storytelling animals – we instinctively learn by the experience of others. My friend ate that plant with the bright red berries and then became very ill – lesson: don’t eat from that plant. This is a type of heuristic, a mental shortcut that humans evolved in order to make quick and mostly accurate judgments about their environment. From an evolutionary point of view it is probably statistically advantageous just to avoid the plant with the red berries rather than conduct blinded experiments to see if it really was the plant that made your friend sick.
Further, the most compelling stories are our own. When we believe we have experienced something directly, it is difficult to impossible to convince us otherwise. It’s just the way humans are hardwired.
Understanding the world through stories was a good strategy in the environment of our evolutionary history but is far too flawed to deal with the complex world we live in today. In fact, the discipline of science developed as a tool to go beyond the efficient but flawed techniques we evolved. Perhaps, for example, your friend became ill because of the raw eggs he consumed earlier in the day, and the plant had nothing to do with it. Evolutionary pressures favored a more simplistic approach to nature, one that tended to assume that apparent patterns were real.
In today’s modern society we are confronted with a dizzying array of apparent patterns and using the simple rules of thumb we evolved to deal with them is not adequate. Whether or not a treatment works for a symptom or disease is a good example. Symptoms tend to vary over time, some may spontaneously remit, and our perceptions of symptoms are susceptible to a host of psychological factors. There are also numerous biological factors that may have an effect. If we are to make reliable decisions about the effects of specific interventions on symptoms and diseases we will need to do better than uncontrolled observation, or anecdotes.
The primary weakness of anecdotes as evidence is that they are not controlled. This opens them up to many hidden variables that could potentially affect the results. We therefore cannot make any reliable assumptions about which variable (for example a specific treatment) was responsible for any apparent improvement.
Here are some specific factors that make it difficult to impossible to reliably interpret anecdotal medical evidence:
Regression to the mean: This is a statistical phenomenon whereby any extreme variation is likely to be followed by a more average variation – by chance alone. Many diseases have variable or fluctuating symptoms – good days and bad days, or periods of exacerbation followed by periods of relative relief. If a person seeks out a treatment when their symptoms are severe, by chance alone this is likely to be followed by a period when the symptoms are not as severe.
Most illnesses are self-limiting: The old saying goes that if you don’t treat a cold it will last for seven days, and if you treat it it will last for a week. Most ailments get better or improve on their own, therefore most treatments will be followed by symptom resolution even if the treatment has no biological effect. More broadly, all illnesses have a natural history, a course they typically follow over time. In order to know if a treatment is affecting that course it has to be compared to patients who are not treated, or receive a different treatment.
Multiple treatments: Often people will try multiple treatments for a disease or ailment making it impossible to tell which treatment had a beneficial effect, if any. Multiple treatments may be taken all at once, or sequentially. For example, a person with a long term illness (but one destined to have a period of relative relief) tries treatment A without effect, then treatment B without effect, then treatment C which is followed by improvements in their symptoms. They then credit treatment C, recounting how multiple other treatments had failed. However, since the person was trying some new treatment most of the time at any point that their symptoms improved there would be a treatment they could credit with that improvement.
Dead men tell no tales (the problem of reporting bias): Cancer survivor groups do not contain people who died of their cancer. Those who die of a disease are not around to give their anecdotes. There is therefore a built in reporting bias. Also, those who feel they were helped by a treatment are much more likely to boast about it than those for whom there was no apparent benefit. People like to tell the tale of the miracle cure they found and had faith in, despite the skeptics and naysayers – but their vision paid off as the treatment worked for them. People have no motivation to recount their experience with the novel treatment that did not work. Further, patients who feel they are being helped by their doctor or practitioner are more likely to return. Those who feel the treatments are not working may not come back at all to report the treatment failure.
Confirmation bias: It is a well-described psychological phenomenon that we tend to seek out and remember information that confirmed what we already believe, or want to believe, and we avoid, forget, or explain away disconfirming evidence.
Vague outcome measures: Good clinical trials use objective outcome measures – those that are binary (like death or survival), quantitative (like a blood level), or are based upon a specific physical finding. Subjective symptoms do not make good outcome measures because they require that judgments be made, and that introduces yet another variable. Should you count those mild sniffles as having a cold? If you are taking a remedy that you think will help you avoid colds you may dismiss those sniffles and report (and even remember) that you did not get any colds while taking the treatment.
The Placebo Effect: The placebo effect is actually a host of many effects that give the appearance of a response to an inactive treatment. These factors include many of the things I listed above, but also other variables that may alter health outcomes or symptoms. See here for a more complete discussion.
The Fallibility of Human Memory: Medical students quickly learn that one of the biggest challenges in taking a medical history is that people are poor historians, which a polite way of saying that human memory is terrible. Anecdotes largely depend upon an individual’s memory of their illness and treatment. This introduces many new variables. There is, for example, a tendency for people to conflate different events in their memory into a single event, or to combine details from various events. There is also a tendency for details to evolve over time to make a story more clean and profound. So people may, in their memory, exaggerate the severity of their symptoms prior to treatment, exaggerate the response to the treatment, clean up the timeline of events so that improvement began very soon after a treatment (rather than before or long after), forget other treatments that were taken, distort what they were told by their various health care providers, etc. I have had countless opportunities to compare a patient’s memory of their illness and treatment to the documented medical records, and the correlation ranges from poor to completely wrong.
For these, and other reasons, scientists have learned not to trust anecdotal reports – or rather to have a realistic assessment of their reliability. This is why it always strikes me as profoundly naive when anyone presents anecdotal evidence as if it is compelling, or even argues that anecdotes should be relied upon as valid evidence.
We also have history to inform our opinions about anecdotes. Western practitioners relied upon the humoral theory of health and illness for thousands of years. Apparently thousands of years of anecdotal experience did not inform them that their treatments were worthless or harmful. Dr. Abrams became wealthy by selling a machine to diagnosis and treat ailments. His devices were widely used, with millions of people swearing by their effectiveness. It worked for them, and their experience was unshakable. When Abrams died it was discovered that his machines (previously protected from inspection) were filled with useless random machine parts. At the turn of the century radioactive tonics were popular, until prominent proponents began seeing the ill effects of radiation poisoning.
The point of these examples is that anecdotal evidence led many people to conclude that these interventions worked. They are useful examples because they are no longer accepted, humoral theory was replaced by scientific medicine, Abrams devices were dramatically exposed, and radiation therapy is directly harmful. But for treatments that are not directly harmful (and least not in an obvious way) or where there is no “man behind the curtain” to dramatically expose, all we have are the anecdotes – and clearly they are not reliable.
Even in mainstream medicine we have learned to distrust anecdotal evidence, even our own. The history of medicine is strewn with treatments that seemed to work but then were abandoned when scientific evidence showed otherwise. The classic example of this is mammary artery bypass for cardiac angina – it seemed anecdotally to work, but it didn’t.
But should anecdotes play any role in medical evidence? Yes, but a very minor and clearly defined one. Anecdotes, with all their weaknesses, are real life experience. It is possible that a treatment does in fact work and personal experience may be the first indication that there is a meaningful biological effect in play. But here are two limiting factors in how anecdotes should be incorporated into medical evidence:
The first is that anecdotes should be documented as carefully as possible. This is a common practice in scientific medicine, where anecdotes are called case reports (when reported individually) or a case series (when a few related anecdotes are reported). Case reports are anecdotal because they are retrospective and not controlled. But it can be helpful to relay a case where all the relevant information is carefully documented – the timeline of events, all treatments that were given, test results, exam findings, etc. This at least locks this information into place and prevents further distortion by memory. It also attempts to document as many confounding variables as possible.
The second criterion for the proper use of anecdotes in scientific medicine is that they should be thought of as preliminary only – as a means of pointing the way to future research. They should never be considered as definitive or compelling by themselves. Any findings or conclusions suggested by anecdotal case reports need to be later verified by controlled prospective clinical studies.
Understanding the nature and role of anecdotes is vital to bridging the gap between the proponents of science-based medicine and believers in dubious or sectarian health practices (as well as the public at large). In my experience it is often the final point of contention between these two camps.
It is interesting to note that the scientific community has long ago made up its collective mind about the weaknesses and role of anecdotes. Logic and the lessons of history speak very clearly on this issue. But there are forces at work today that want to turn back the clock on scientific progress – they want to bring back anecdotes as a reliable source of medical evidence, essentially returning to the pre-scientific era of medicine. In some cases this is done out of frustration – that controlled scientific data has not validated a prior strongly held belief. In other cases it seems to be a calculated attempt to lower the bar of evidence to admit treatments that have not been validated by solid scientific evidence. In either case, this is not in the best interest of the health of the public.