In the three prior posts of this series I tried to analyze some of the defects in the randomized clinical rials (RCTs) of homeopathic remedies for childhood diarrhea. The first entry showed that the first two RCTs’ (done in Nicaragua) methods could not produce a meaningful result because of the way the RCTs were set up (methods.) The second entry showed that the results obtained in the first two trials were meaningless clinically even if assumed to have resulted from more legitimate methods. The same applied to the third trial in Nepal, analyzed in the third entry.
This entry will suggest that the authors’ fourth paper (Jacobs J, Jonas WB, Jimenez-Perez M, Crothers D. Homeopathy for childhood diarrhea: combined results and metaanalysis from three randomized, controlled clinical trials. Pediat Inf Dis J, 2005;22:229-234.)- a meta-analysis (MA) of the data from the three RCTs resulted in conclusions equally as meaningless as those of the three trials.
The MA authors – several of the same workers from the three RCTs – begin by agreeing that the data from the RCTs, taken individually, were of borderline significance:
In our previous three studies, we evaluated the use of individualized homeopathic treatment of childhood diarrhea … The results of the two larger studies (n = 81, n = 116) were just at or near level of statistical significance. Because all three studies followed the same basic study design , […] we analyzed the combined data from these three studies to obtain greater statistical power. In addition we conducted a meta-analysis of effect-size difference […] to look for consistency of effects.
MAs and systematic reviews (SRs) are the two consensus methods for summarizing data from multiple individual studies. The inclusion and search methods of RCTs for SRs and MAs are similar, but the objectives of the two are a bit different, as are the forms of the reports. In SRs, the results are summarized in more in narrative form, whereas in MAs the data are treated mathematically and the results are defined in statistical terms. Thus authors of SRs are freer to speculate on the degree of confidence that a method is effective based on what is shown by the numbers of positive and negative RCTs collected. Authors of MAs usually limit their comments to what the mathematical formulation of the summarized data show.
I am not a statistician, and will not comment so much on the mathematical aspects of the MA in question here, but will point out that 1) themethods used were standard and reported credibly, and 2) the problems found in the RCTs contribute to invalidating the conclusions of the MA. A common warning to authors of MAs is that the outcome of any MA depends directly on the reliability of the individual RCTs.
But what did this MA show? The primary outcome measure was the number of days from entry until the presence of no more than 2 stools per day for 2 days. 247 subjects entered the study and 230 completed the study through to the final end point (nearly equal numbers in each arm dropped out or had incomplete data.) There was an overall 18.5 percent difference between the homeopathic treatment and the placebo control groups, with the reduction from 3.8 days in the control group to 3.1 days in the treatment group. (P=0.008). The P value appears to impart a high significance to the result. But similar problems that played out in the RCTs played here in the MA as well.
Examination of the Kaplan-Meier plot showed that at each day after day 0 the homeopathy group showed a less likely presence of diarrhea than the control group. (Kaplan -Meier curves are constructed to show probabilities, not actual events.) But at day 1 (24 h) the difference was about 15percent, the same at day2, but at 3 to 4 days, the period of the primary measure, the difference was greatest at about 25-30 percent. By the fifth day the difference between the two groups was down to about 10 – 15 percent. In other words, the greatest statistically significant period in the study happened to occur at the time of the selected end point, but was much less at the other points. There is no evidence of data manipulation here, just an odd finding. I cannot account for it unless that difference is a normal characteristic of such studies. Odd.
But shedding the arithmetic and statistical wording, and describing what actually occurred, even at maximum difference between homeopathy and controls, was a difference between having 3 (or slightly more) stools per day and no more than 2 stools per day for 1 day or so at only one of four measured periods of the study. The calculated difference in terms of days to the end point was slightly more than a half a day. Most patients and most family members would hardly be aware of such a small difference. A sort of homeopathic difference.
In addition, the authors sought to find any differences between the groups that could affect outcome such as age, size, that occurred during assignment or randomization, and found that there was a difference in assignment that favored the homeopathy groip (P = .025.) Not a great difference, but an indication of how much chance differences can contribute to study outcomes, that combining studies and increasing numbers does not always even out random imbalances. This difference also suggests that sometimes there can be hidden biases in apparently well carried-out studies.
So the MA showed the same kinds of outcomes as the three individual studies did, but with a smaller P value from the larger number of subjects via the combining of the data. Pretty much what would have been expected. But for a minor symptom, and for results that should not be extended to the general population, since the most severely affected children were hospitalized and excluded from the study.
However, in the political system of homeocracy, rules change at the power of thought, scientific method bcomes a “living, breathing document”, the principles of which are malleable to fit circumstances. Thus, despite the minimal findings from this series of studies, the minimal boost in significance from the MA, the authors conclude again that homeopathy might be added to the therapy of childhood diarrhea because although the advantage is small, the treatments are harmless, the world-wide clinical problem is great, and so homeopathy would be an advantage to public health.
One can still wonder how papers with such comments pass editorial and peer review – but then, papers with famous names tend to get attention (Wayne Jonas is former Director of the Office of Alternative Medicine and a homeopath.)