Recent comments on homeopathy again resulted in references to the 1994 Pediatrics paper by Jacobs et al on use of homeopathic remedies for childhood diarrhrea. The authors of that paper concluded from their blinded study that homeopathic remedies, tailored to the individual infants and children, were effective in reducing the number of diarrhea stools and thus in shortening the illness. The paper has been widely referenced and reported to have proved homeopathy efficacy, and the critiques have been argued by homeopaths as irrelevant, as has been done by Mr. Ullman.
In this short series I want to recount my experience with the report, its predecessor, and its two major sequellae, as well as its effect on systematic reviews of homeopathy. I will begin with a description of the first two studies, concentrating on their methods. Then I will discuss the results of the 1994 Pediatrics report and the authors’ interpretation of the results. Then (I hope last) I will discuss the third paper which the authors claimed supported findings of the first two and the meta-analysis which combined data from all three. If you are imagining why this series interests me, and imagining the worst, you are probably right. The sequence will help to reveal how some of the information in “holistic” and “alternative” systems become published, and despite critiques and disproofs through a sort of systematic deconstructions, still develop wheels of their own and enter the fund of general knowledge seemingly forever. Or, at least for several decades, until a social belief switch is finally turned off.
These effects have in turn, political and economic effects, that are of more urgent concern now as promoters lobby to have sectarian methods covered under insurance and federal medical plans. Probably as important, is that these reports will illustrate a different way of looking at the literature of implausible propositions. To destroy the suspense and let you in on the idea, the idea goes something like this: We usually look at positive reports on quacky methods to find falsification or fatal errors or hidden or absent data that disprove the thesis. Those are satisfying when present. But there is an alternate route; disqualify or at least diminish the meaning of a study because of the way it was set up, carried out, and/or reported, and the way that the worthless outcome data were subsequently used (the garbage in side.) The garbage pile as it accumulates, results in higher degrees of unbelievably – as it lacks both authenticity and validity. The focus must then turn to the statistical methods and how they are used to (mis)interpret the pile of data.
What often happens here is that the authors apply statistics designed to analyze data that are collected from experiments that are carried out in uniform ways, with reliable methods and instruments, in order to distinguish or characterize plausible and measurable phenomena. When one measures imaginary phenomena, then the results do not tell anything about the phenomena, but about the experiments themselves – and the errors in the experimental and measuring system. That is where I’m going with this. My analysis will show that the data collected were so heterogeneous and random, and the measuring so subjective, that the results were meaningless (also garbage.)
The affair began with Jacobs’s report of a pilot study published in the British Homeopathic Journal in 1993 (1). I was unaware of this report until I read the next one in Pediatrics, 1994. (2). I, with co-author Bill London, a health educator with the Natl. Council against Health Fraud wrote a five page analysis of the second paper. That came about when a nursing educator in the Pharmacy dept. at the teaching hospital where I taught medical oncology caught me in a hallway and asked me to examine the report which concluded that homeopathy worked for childhood diarrhea. She and our head pharmacist knew of my interest in medical quackery,. She did not know how to explain the report to her students. So I bit on it.
It was more difficult than I thought, for several reasons. First, I was familiar with homeopathy, but not as familiar with its modern practice. I learned that homeopaths use a computerized system for selecting remedies. More problems came after those.
I contacted the editor and asked if he would consider publishing a critique, considering the potential impact – treatment of millions of children in underdeveloped nations, and the implausibility that homeopathy worked. The editor was prompt, replying yes, partly because – as we later learned – he felt his job was on the line for having accepted the Jacobs manuscript, so vigorous was the negative response from readers.
As for the trial setups, we assumed they were the same for all three trials, although the amount of details reported varied. We found unanticipated problems. The diagnoses – other than diarrhea – on which remedies were selected were criteria based on symptoms only. Here are sample criteria from the table. They were divided into those of “mind,” “general,” and “stool” (quality).
Mind: Great anxiety and restlessness, fearful,
Capricious; irritable; quarrelsome, nothing p[leases, striking out, moaning, frenzied…
Slow, lethargic, weakness of memory (in a 1-2 year old?)
Diarrhea after mental excitement. Fidgety and restless.
Irritable, indifferent, weeping.
Stools: Acrid, burning, excoriating, worse after midnight,
Green, slimy, offensive, chopped grass, … smelling like rotten eggs, …
Slimy, scanty, lienteric, bloody. Acrid, corrosive, frothy, yellow. Sour odor.
Profuse, frequent, gushing, painless, watery. Bloody with green mucus, offensive. Rectal
prolapse. Exhaustion after stool…
Worse at night, after milk, involuntary, sudden expulsion. Worse 5 AM (sic) Red ring around
anus. Offensive, acrid stools. Painless, sour; thin; watery. Odor of rotten eggs.
General: [only the first criterion for each collection of criteria for brevity)
Prostrate,
one cheek red, the other pale;
Chills/heat;
gagging/ retching;
Cold sweat face and feet
Each combination (of each category – first to fifth – of the above called for a different homeopathic remedy.
Not mentioned was how each criterion combination / remedy was determined. Apparently by prior “provings.” But “proving” have never been proved to our knowledge, by repeated trials – blinded or not. So already we are in deep trouble. The criteria overlap, are subjective and made by a second party – often a relative – in answer to specific questions. Each answer depends on memory for the prior 1-2 days, powers of observation, sense of small, frequency of diaper change, and so forth.
Thus we have:
- A glut of information, all highly subjective and subject to multiplicity of error sources.
- A data base not proved for validity or consistency.
- Overlapping criteria (admitted by the authors).
The next problem is the selection of timing. The initial visits, had to have been after 2 days of illness, but there was no specification of at which point the questions were asked. Thus, the quality of the stools, the state of “mind“ and the general appearance could have been at any time after the inclusion period. Given the large variation in those qualities from hour to hour and day to day, the chosen remedy was based also on the time of the answer, a “snapshot in time” upon which the next few days’ remedies were chosen and given. One can imagine the wide descriptive differences obtained, yet all were treated as homogeneously obtained. Yet these data were treated with statistics as if those descriptions and remedies were tightly controlled.
The potency of all selected remedies was 30C…that is, 1X 10/\ -300. How come? We have not even begun to deal with the problem of the homeopathic dosage system yet. Don’t need it for this part of the discussion. In order to grasp the significance of what the homeopaths did in these clinical trials, I got the idea yesterday to analogize the problem in physics terms. With apologies to my physics profs and associates, I tried it this way (and please feel free to critique this as I have not run this by anyone else…instant peer review!)
I imagine a physics experiment done under similar conditions. Test the influence of variable doses of electricity on a series atomic particles – electrons, protons, whatever – with their energies and “spins” varying to hundreds of variations, with the “dose” of the electric field chosen from a previously set of criteria which had been tested only once or twice for effects on a different set of particles – the effects recorded and entered into a book (The Organon) and a modern computer data base. Then select a random “dose” or power, charge, etc., to be applied to each test particle at random times after the particle had traveled a minimum distance from its source. Then measure the effect on each particle at four defined times, record all of the values as a group at each time, then analyze them with a statistic. If a pattern shows up, would it mean anything?
I guess homeopaths are right in a sense when they claim that standard science cannot measure what homeopathy does. Nothing can. Especially as their computer cannot organize the data, laden with close to randomized bits.
Homeopaths are caught on the point of their own petard here. They claim to be able to select remedies based on previous proving as recorded in “The Organon” in its several iterations, and the information placed in a computer program based on that information. The information going in is in reality “disorganized” or “unorganized” to play on a few words, and no amount of excusing or rationalization can organize it.
Thus, the outcomes are nothing more than random bits of information, even if carefully measured. And therefore, any emerging patterns do not indicate efficacy or harm, but only normal variations of random patterns. They measure only the noise in the system.
Enough mental exercise for this time. Next I’ll look at the second paper’s outcome data and what the authors did with it, and move on to the third paper’s adventure in the journal system, then on to the meta-analysis.
REFERENCES:
- Jacobs J, et al, Homeopathic treatment of childhood diarrhea. Brit Homeop J, 1993;82(2):83-86.
- Jacobs J et al. Treatment of Acute Childhood Diarrhea with Homeopathic Medicine: A Randomized Clinical Trial in Nicaragua. Pediatrics, 1994;93(5):719-724.
Posts in the series: