The previous post of this series analyzed the results of the 1994 Pediatrics paper purporting to show a statistically significant effect of homeopathic preparations on acute childhood diarrhea in a population in Nicaragua. That clinical trial followed a pilot study that also had shown a small but statistically significant effect of homeopathic remedies.
A moment here for explanation as to why I am going through these old studies. Reports like the four or five in this series made headlines. They are also so well cloaked in manipulated data and overdrawn conclusions that press and even academicians accept their conclusions – and even overdraw more. This is still going on.
Over the past thirty years some of us informally and gradually developed semi-systematic ways of analyzing these increasingly scientific-appearing claims of sectarians (sCAMmers.) Errors, inconsistencies and falsifications we recognize now were not so obvious decades ago. SCAMmers developed imaginatively new methods as their fields progressed. We in the science-based or knowledge based medicine field have been trailing along, detecting their tricks and twists as they developed, and like street sweepers behind horses, picking up their excrement (metaphor to force attention.) Yesterday’s lucid post on the latest acupuncture study by Steve Novella exemplifies this expertise (no offense intended.)
In summary, what the previous posts recounted was our response to the first pediatrics article, pointing out the heterogeneity of the trial data and the misapplication of statistics, and the invalid treatment selection process.
This third clinical trial (Jacobs J, Jimenez , Malthouse S,
Chapman E, Masuk M, Jonas W. Homeopathic Treatment of Acute Childhood Diarrhea: Results from a Clinical Trial in Nepal. J Altern Compl Med,2000;6(2):131-139.) was claimed by the authors to have added verification to the first two. The Nepal trial had the same trial setup and subject assignment, the same treatment selection method, and same treatment schedule.
The trial was stated to be carried out months before publication of the second trial, as a replication, and to correct for certain odd findings in the second trial. One objection we raised to the second Nicaragua trial was that most of the difference found between the treatment and control groups might have been due to finding more stool exams testing positive for bacterial or protozoal pathogens in the homeopathic treatment group than in the placebo group. If a significant number of subjects had been treated extraneously with antibiotics (common in Latin America), that action could have been responsible for the difference in diarrhea resolution between the two groups.
In this third trial, all subjects’ stools were cultured at entry, and the infection/infestation rates were reported as nearly equal. In addition, “Infrared spectroscopic analysis of bottles chosen randomly from both groups showed no evidence of contamination of samples by inorganic or organic substances.” The number of bottles tested was not stated. But since the identified infection rate was equally distributed, presence of antibiotics would not likely have been significant.
But…there seemed to be a difference between the reported end points of the two studies. The Nepal study abstract stated two end points under results – at least the wording and the text graph presentation were not identical. In the abstract and the methods section, is found the following wording: 1) The mean number of stools per day over the entire 5-day treatment period was 3.2 for the treatment group and 4.5 for the placebo group (t = 2.30, df = 123, P =.023; and 2) A Kaplan Meyer survival analysis of the duration of diarrhea, which included data from all patient visits, showed an 18.4% greater probability that a child would be free of diarrhea by day 5 under homeopathic treatment (P =.036.)
In the methods section authors stated, ”The duration of diarrhea was predefined as the time until there were two consecutive days with fewer than three unformed stools, the same primary outcome measure as in our previous [Nicaragua] study. Re-examination of the Nicaragua study’s end point showed the same wording. However, in the Nepal study, because about ten percent of the subjects (five in each group) did not complete this study, an intention-to-treat analysis (Kaplan-Meyer) curve was constructed, showing the likelihood of the primary outcome at points in time (daily in this case.) Although a small difference was again obtained in this study, the data were treated differently and the end point was not really the same.
One easy way to see this difference – sorry I could not transfer the actual graphs from the PDF – is to state what was plotted on the coordinates. In both studies the abscissa was the number of days, (0-5). In the Nicaragua study the ordinate was mean number of unformed stools each day. In the Present Nepal study, the ordinate was the “Probability of Diarrhea.” These are not the same, although the graphs appear similar, the data were treated differently. In the Nicaragua study, the difference between arms could be seen to be clinically insignificant even though statistically significant because at maximum difference, the difference amounted to one unformed stool per day (3 vs.2., equal to one stool every 12 hrs opposed to one stool every 8 hrs. ) That difference would not have been enough to warrant adding homeopathy materials to the treatment of childhood diarrhea.
In this Nepal study, the homeopathy and placebo groups were equivalent for the first 3 days when diarrhea stools were most frequent. The curves broke at 4 and 5 days, at which the differences were about 20 percent (40 percent vs. 60 percent) different; the homeopathy group would have been 20 percent more likely to be free of diarrhea at the specified time. Not much difference – and late, when it counted least…that is, if the results were not due to chance or some extraneous factors.
In the previous Nicaragua study, authors listed three other secondary outcomes (days to 50% improvement, days to first formed stool, mean diarrhea index) that were not listed in the Nepal study. In the Nepal study, at least one other secondary end point was listed that was not listed in the Nicaragua study (mean number of stools per day over the entire 5-day period – 3.2 for the treatment group and 4.5 for the placebo group.) The different presentations made it difficult to compare the two studies.
In summary, although the results were found to have positive outcomes for the homeopathy treatment group in both studies, the primary end points were calculated and presented differently, and secondary end points of both studies were not the same.
Yet another problem did not require statistical expertise to see. Despite randomized assignment, there was a significant age, height, and weight difference between the two groups (P = .07, .03, .05, respectively.) The assignment procedure resulted in older, taller, and heavier subjects assigned to the homeopathy group than to the placebo group, and the difference was statistically significant. Age is significant because older, larger children recover from diarrhea more quickly than do infants and smaller children. The difference in assignment was described as inadvertent, and a regression analysis correction was applied.
The results were obscure to me but here is the point. If a random assignment could result in a significant imbalance for age (weight, height) and that difference is assigned to chance, then why is a similar statistically significant difference in treatment outcome due to the treatment, and not to chance? The size of the assignment difference was of a similar order of magnitude in both instances. Nevertheless, the authors claimed, as they had done in the Nicaragua studies, efficacy from administration of homeopathic preparations in childhood diarrhea.
As in the prior study, the difference may not have been from effectiveness of the remedy, but from the vagaries inherent in doing clinical trials on ineffective remedies with diagnoses and end points that are as subjective as objective. There was just too much noise in the system.
One can still find references to these studies as positive in homeopathy commentaries. Not only were the trial setups done in a way to invalidate any but quite large differences in study arms, but the results were claimed to be a public health benefit in severe diarrhea despite the fact that severe diarrhea subjects were excluded from the studies.
Another sour note. A few years before this publication, I reviewed a similar paper that claimed similar results but that reported a positive end point that was negative in the Nicaragua report, and that reported as negative the end point reported as positive in the Nicaragua report. In other words, the end points cancelled each other, yet the authors (same? different?) reported the results as confirming the Nicaragua trial. I cannot be sure this Nepal study is the same, but the paper was rejected by at least one major journal. This one with a different end point presentation showed up in a sectarian journal. Just wondering.
I see eyelids that are still open are drooping, so the analysis of the authors’ meta-analysis of the three studies will wait until next posting. That will likely be shorter, as there will be fewer points made. The fourth study, diarrhea in Honduras, was already adequately done by David Gorski.