A newly published meta-analysis of studies looking at acupuncture for symptoms resulting from natural menopause (not drug or surgically induced) by Chiu et. al. is entirely negative. That is not what the authors or the press release conclude, however.
This disconnect between the study results and the interpretation of those results is a persistent problem in medicine generally to some degree, but is endemic and profound within the CAM (complementary and alternative medicine) culture. Acupuncture in particular is promoted almost entirely based on this type of misinterpretations – the kind that can magically turn negative studies into positive studies.
In the abstract the authors conclude:
This meta-analysis confirms that acupuncture improves hot flash frequency and severity, menopause-related symptoms, and quality of life (in the vasomotor domain) in women experiencing natural menopause.
Let’s take a close look at the results, however. Indeed, when comparing acupuncture to no treatment controls there was a significant decrease in subjective symptoms in the pooled data. Outcomes were hot flash frequency, hot flash severity, other menopausal symptoms, and quality of life. Some of the included studies were large controlled trials, which the authors used to argue that their results are valid. They also point out that their results showed heterogeneity and lack of publication bias.
There are two major problems with the results, however, that contradict the authors’ conclusion, one moderate and one fatal. The moderate problem is that there is a lack of dose response. The authors write:
The effect size of acupuncture on hot flash frequency was not significantly associated with the number of treatment doses, the number of sessions, and the duration of treatment in weeks.
The same was true of hot flash severity. We like to see a dose-response effect to help confirm that the treatment effect is real. You might argue that acupuncture does not work that way, that even a brief treatment is sufficient and there is an early plateau in effect. This might be true (in which case longer sessions are unnecessary), but this argument amounts to special pleading, and at the very least implies caution in concluding the treatment works.
The second problem with the data, however, directly contradicts the authors’ conclusions:
Arguably, our findings showed that sham acupuncture could induce a treatment effect comparable with that of true acupuncture for the reduction of hot flash frequency. This result is in line with previous reviews. A compelling finding from a previous systematic review concluded that approximately 60% of RCTs revealed that sham acupuncture was as efficacious as true acupuncture, especially when superficial needling was applied to nonpoints.
We found insufficient evidence to determine whether acupuncture is effective for controlling menopausal vasomotor symptoms. When we compared acupuncture with sham acupuncture, there was no evidence of a significant difference in their effect on menopausal vasomotor symptoms. When we compared acupuncture with no treatment there appeared to be a benefit from acupuncture, but acupuncture appeared to be less effective than HT.
Both reviews are essentially showing the same thing – when you compare acupuncture to sham acupuncture, there is no difference in outcome. The spin on these results, however, is entirely different. The Chiu review goes on to speculate about why sham acupuncture might also “work.”
We have pointed out this invalid logic many times. Especially when dealing with subjective symptoms, the only comparison that tells us anything reliable is a blinded comparison between a treatment and a control. Unblinded comparisons (such as between sham acupuncture and no treatment) are essentially useless, as subjective outcomes will be contaminated with placebo effects.
You also have to consider the quality of the study. Acupuncture studies in particular have a problem with unblinding (the subjects figuring out if they had real or sham acupuncture). The Chiu review summarized the rigor of the studies they reviewed, many of which had problems with allocation concealment, blinding of outcome assessment, and selective reporting. Such flaws easily explain the 40% of studies that show some small difference between treatment and control.
This pattern is typical of the acupuncture literature. The best quality studies consistently show no difference in blinded comparisons between treatment and control groups. The variables that define acupuncture – sticking needles into acupuncture points – do not make any difference to the outcome, therefore acupuncture (by any reasonable definition) does not work. The therapeutic ritual surrounding acupuncture has predictable placebo effects and nothing more.
A possible explanation for the dramatic difference in interpretation of the same data between the Chiu and the Cochrane reviews could be in a conflict of interest in the Chiu review. Although they disclose no conflicts of interest, all the authors are from Taiwan. A 1998 review found:
Research conducted in certain countries was uniformly favorable to acupuncture; all trials originating in China, Japan, Hong Kong, and Taiwan were positive, as were 10 out of 11 of those published in Russia/USSR.
From this data it is reasonable to conclude that being from Taiwan is a conflict of interest for acupuncture studies.
Yet again we see that the clinical data with acupuncture studies is essentially negative. In this case acupuncture does not work for menopausal symptoms. Proponents of acupuncture, however, turn the logic of science and clinical trials on its head by bizarrely concluding that no difference between treatment and control means the control works also (rather than the usual interpretation that the treatment does not work).
Believers apparently buy into this invalid logic. For example, the website Science 2.0 (a straight-up science news site) has the headline: Acupuncture Works To Reduce Menopause Hot Flashes – Meta-analysis. This is followed by a gullible article that accepts the authors’ biased conclusions without any critical analysis. The comments are also revealing, one stating the CAM party line very well:
You actually are quite uninformed here as to what “placebo” is. It is a mass misconception that “placebo effect” means something doesn’t work. The placebo effect actually proves that the body has a remarkable ability to heal itself. This is most relevant in the field of acupuncture – sadly, to fit acupuncture into the realm of “double blind studies” (which you would understand are horribly flawed if you spent any time in medical schools and in research classes (I have for instance) they had to come up with “sham acupuncture” which is completely flawed, because there’s no way to fake it.
No difference between treatment and placebo does mean that, as far as the clinical trial is concerned, the treatment has no specific effect. Sham acupuncture nicely controls for one aspect of acupuncture – alleged acupuncture points. At the very least we can conclude that acupuncture points have no basis in reality, which undercuts the traditional philosophy of acupuncture (at least as it is received today).
Other studies show that needle insertion, or insertion to depth, or elicitation of the alleged “de Qi“, also have no effect. You can actually “fake it” by using opaque sheaths and dull needles or even toothpicks that are not inserted, but neither the patient nor the acupuncturist can tell the difference.
Again, at the very least from all the data we must conclude that coaching a completely untrained person to poke the skin randomly with toothpicks, while putting on a halfway decent show, is just as effective as individualized acupuncture from a trained and credentialed acupuncturist with decades of experience. Therefore there is absolutely no basis for acupuncture philosophy, and no justification for training or credentialing of acupuncturists. Whatever they are teaching seems to be worthless.
A reasonable person can only conclude that acupuncture does not work, and that all the clinical research consistently shows that acupuncture conveys only illusory and nonspecific placebo effects for subjective symptoms.