A recent meta-analysis of acupuncture studies for chronic pain by Vickers et al is getting a great deal of press. The authors’ conclusions are:
Acupuncture is effective for the treatment of chronic pain and is therefore a reasonable referral option. Significant differences between true and sham acupuncture indicate that acupuncture is more than a placebo. However, these differences are relatively modest, suggesting that factors in addition to the specific effects of needling are important contributors to the therapeutic effects of acupuncture.
News reports generally reflect this conclusion – acupuncture works, but mostly (although not entirely) through placebo effect, but that’s OK.
I took a close look at the study and find that the authors display considerable pro-acupuncture bias in their analysis and discussion. They clearly want acupuncture to work. That aside, the data are simply not compelling, and the authors, in my opinion, grossly overcall the results, which are compatible with the conclusion that there are no specific effects to acupuncture beyond placebo.
The meta-analysis looked at 29 randomized clinical trials of acupuncture in back pain, neck pain, headache, and osteoarthritis, involving both sham and no-acupuncture controls. The differences between acupuncture and no-acupuncture were large, reflecting an absolute reduction in pain of about a 30% (50% relative reduction). However, the authors acknowledge:
Because the comparisons between acupuncture and no-acupuncture cannot be blinded, both performance and response bias are possible.
In other words – the unblinded comparison between acupuncture and no acupuncture is entirely overwhelmed by bias and completely useless. The no acupuncture control groups involved patients continuing to receive usual care (whatever they were already receiving that was not effective, or sometimes just being told not to get acupuncture). This was not a comparison to any specific medical intervention. In other words, the subjects were aware they were receiving no treatment.
It is curious that the authors would even bother to include such an analysis, but they reveal their purpose in their discussion:
Even though on average these effects are small, the clinical decision made by physicians and patients is not between true and sham acupuncture but between a referral to an acupuncturist or avoiding such a referral.
This is the agenda of acupuncture proponents – to use the non-specific effect of receiving an intervention to promote the use of acupuncture. If a study shows no significant difference between true and sham acupuncture, then they argue that this placebo effect is enough to justify treatment. If the study (or in this case a meta-analysis) shows a small difference, then they use that small difference to justify the conclusion that acupuncture is real (even though the specific effects are negligible) and then use the large non-specific effects to justify the treatment.
Either way, proponents are inappropriately leveraging placebo effects (which are largely biases) to promote a treatment that has an effect size that is very small and, in my opinion, overlaps with no effect at all.
The authors make much of the small effect difference in their meta-analysis between true and sham acupuncture. They summarize their results by saying, if the no intervention group has a pain of 60%, then true acupuncture reduces it to 30% and sham acupuncture to 35%. While this difference was statistically significant in this meta-analysis, it is highly dubious to claim that the 5% difference is clinically significant, or even perceptible. To me this is no difference at all.
The primary difference between my opinion of this data and the authors’, however, is that the authors are quick to conclude that because their data was statistically significant that means there is a real physiological effect (if modest) to acupuncture. This conclusion, however, reflects probable bias, but certain naivete with regards to the reliability of clinical trials. This level of difference is within the noise of clinical trials, which are simply not precise enough to detect such a small difference.
The authors acknowledge:
Similarly, while we considered the risk of bias of unblinding low in most studies comparing acupuncture and sham acupuncture, health care providers obviously were aware of the treatment provided, and, as such, a certain degree of bias of our effect estimate for specific effects cannot be entirely ruled out.
This is the understatement of the paper – a certain degree of bias cannot be ruled out. What the last century of clinical research has clearly shown is that a significant amount of bias is guaranteed. All it takes is a little bias, innocent exploitation of researcher degrees of freedom, and you have your 5%.
The authors acknowledge that there are a couple of studies that are outliers – those by Vas et.al. had effect sizes 5 times that of the average. When you remove these studies the effects are still significant. There is also researcher bias in that the larger the study the smaller the effect size, but when you remove small studies the effects are still significant. They also argue that unpublished studies (publication bias) would be unlikely to cause their results.
However – when you add the effects from outliers, small studies, and publication bias all together I wonder what the total effect is on the data (not even including the fact that it is a certainty the data is polluted in the false positive direction by researcher degrees of freedom). The performance of a meta-analysis introduces yet another layer of potential bias or distortion in the methods of the meta-anslysis itself – how are studies chosen for inclusion, for example, and all the researcher degrees of freedom that apply to any study.
Conclusion
The Vickers acupuncture meta-analysis, despite the authors’ claims, does not reveal anything new about the acupuncture literature, and does not provide support for use of acupuncture as a legitimate medical intervention. The data show that there is a large difference in outcome when an unblinded comparison is made between treatment and no treatment – an unsurprising result that is of no clinical relevance and says nothing about acupuncture itself.
The comparison between true acupuncture and sham acupuncture shows only a small difference, which is likely not clinically significant or perceptible. More importantly, this small difference is well within the degree of bias and noise that are inherent to clinical trials. Researcher bias, publication bias, outlying effects, and researcher degrees of freedom are more than enough to explain such a small difference. In other words – this data is insufficient to reject the null hypothesis, even if we don’t consider the high implausibility of acupuncture.
Further, meta-analysis itself is an imperfect tool that often does not predict the results of large, rigorous, definitive clinical trials. The best acupuncture trials, those that are well-blinded and include placebo acupuncture, show no specific effects.
