Systematic reviews are a good way to see the big picture of what the evidence for a particular medical intervention looks like. However, they are very difficult to do well and often fall prey to their own methodological limitations and biases. Therefore it is even better to look at a systematic review of systematic reviews as a way of looking at the overall pattern of the evidence. As we have discussed many times on SBM, it is that overall pattern that tells us if we are dealing with a real phenomenon or an illusion of the research (or just a placebo effect).
A systematic review of systematic reviews of acupuncture for a variety of indications published in November 2022 gives us an opportunity to look at that pattern. This is the same evidence we have been reviewing here for years, and so I don’t expect any surprises; still, a thorough look at the evidence is instructive. Before I dive into the study, let’s review what acupuncture is and why I am skeptical of its efficacy.
Acupuncture is generally defined as a practice involving inserting thin needles into acupuncture points on the body that correspond to specific physiological functions and therefore can be used to produce specific clinical outcomes. I recently reviewed an acupuncture systematic review for back pain in pregnancy and went over many of the problems with acupuncture research and claims. I also published a detailed analysis of acupuncture with David Colquhoun that was very critical. Here are my main criticisms of acupuncture as a medical practice and belief system:
- Acupuncture points have no basis in anatomy, physiology, or neuroscience and essentially they don’t exist.
- Acupuncture has no plausible or established mechanism, and many practitioners reference “chi” which is a nonexistent magical life force.
- Acupuncturists claim that acupuncture can work for a wide variety of medical conditions that have nothing functionally to do with each other.
- Acupuncturists can’t agree on where alleged acupuncture points are and what they do. Therefore, different studies of the same condition often use different sets of points.
- After decades of research and thousands of studies there isn’t a single clearly established condition for which acupuncture has demonstrated efficacy.
- There is evidence of extreme researcher and publication bias in the acupuncture literature.
In short, acupuncture has all the red flags of fake science and placebo medicine, and there is no legitimate scientific reason to think that it is real. There have been thousands of clinical studies looking at acupuncture for various conditions, most of which are very low quality and even fatally flawed. But when you conduct thousands of studies, especially with some wiggle room in the methods, many of them are going to be positive, even with a completely null effect. We see this, for example, in the homeopathy literature. Therefore, if my analysis above is fair and accurate, we would expect the same pattern of outcome in the acupuncture literature that we see in the homeopathy literature.
One particular feature that acupuncture research shares with homeopathy research is that it is culture-dependent (another red flag). For acupuncture, the effect is even more dramatic. A 1998 systematic review, with an update in 2014, showed that acupuncture clinical trials based in China are 100% positive. Edzard Ernst characterizes this as an “open secret” that data is simply fabricated in China to support acupuncture. Even if a treatment worked with a large effect size, we still would not see a 100% positive rate in the literature. That is essentially slam-dunk evidence for some combination of publication bias and researcher bias. Of course, these biased Chinese acupuncture studies contaminate all systematic reviews of acupuncture (unless they are systematically excluded, which I have never seen).
Further, as we saw with my most recent review of acupuncture for back pain in pregnancy, even systematic reviews which are clearly negative are presented as if they are positive. The current reviewers seem to take the author’s conclusions at face value, which is highly problematic when dealing with acupuncture.
So – taking all these caveats into consideration, what does the current review of reviews show? It’s actually amazing how weak and negative the evidence is, given all the biases in the literature. Here is the author’s overall big picture take on the evidence:
Despite a vast number of randomized trials, systematic reviews of acupuncture for adult health conditions have rated only a minority of conclusions as high- or moderate-certainty evidence, and most of these were about comparisons with sham treatment or had conclusions of no benefit of acupuncture. Conclusions with moderate or high-certainty evidence that acupuncture is superior to other active therapies were rare.
In other words – most of the acupuncture evidence is crap. This is another pattern we see commonly with fake treatments, and we have to seriously ask – after decades and thousands of studies of acupuncture, why do a low-quality study at all? The methods and technology for high quality acupuncture studies exist. But of course, high quality studies are frustratingly negative, which may be why there are so many low quality studies still being performed.
Of the 56 health condition reviewed here, only 4 had high-quality evidence. Of those four, one was negative – for assistive reproduction, essentially there were no increases in the chance of success with implantation. One, dealing with stroke recovery, used electroacupuncture, which is not acupuncture but rather electrical nerve stimulation. This intervention also included conventional interventions. Meanwhile, another review of stroke not including electroacupuncture was negative.
The other two that involved acupuncture with positive results were for shoulder pain and fibromyalgia, and they both had low numbers of studies. There wasn’t a single indication which had a high number of high quality studies showing a positive effect. The fibromyalgia review contained only 8 studies, two of which were electroacupuncture, with mixed results. Even though some of these studies compared acupuncture to acupuncture on the “wrong” points, there was no agreement across studies which were the correct and which were the wrong points, rendering the comparison meaningless.
The review of shoulder pain was actually part of a larger review looking at a variety of musculoskeletal conditions. This suffered the same problem as the fibromyalgia review – low number of studies, and heterogeneity in terms of controls and the alleged acupuncture points used. Risk of bias in these studies also tends to be high. And again, none control for the clearly demonstrated publication bias in Chinese studies.
Given that there are several thousand published clinical trials looking at acupuncture, over decades, if acupuncture actually worked for anything I would think that by now we would be seeing a clear signal in the data – but we are not. As this review demonstrates, most of the evidence is low quality, and almost none is deemed high quality evidence. Only two subsets had high quality evidence of acupuncture showing an effect, and these suffered from small numbers and methodological issues that were overlooked by the reviewers. Further, if you do enough studies for enough conditions, you’re going to come up with some positive evidence just by chance alone, especially if you throw in a little publication bias.
The overall pattern of the acupuncture literature, therefore, is clearly negative. It is consistent, I would even argue highly predictive, of a null phenomenon. We see this same pattern with other scientifically implausible claims, such as ESP and homeopathy. It’s likely not a coincidence.