In yet another round of science by press release, a particularly unimpressive acupuncture study is making the rounds of the major news outlets proclaiming that acupuncture works. I guess that is a sort-of answer to my title question – why are so many scientifically worthless acupuncture studies being done?
Let’s take a look at this particular study to see why it is so weak. All I have to go on is the press release, since the study is not published. It was presented at a scientific meeting – which is legitimate, I just don’t have access to it. (The bar for publication in a peer-reviewed journal is much higher than presentation at a meeting, and there may, in fact, be changes to the text prior to publication.) But we can still say a great deal about this study from the information provided.
Acupuncture for Hot Flashes
The study looks at acupuncture vs Effexor for the treatment of hot flashes and other symptoms in women who have had breast cancer and are on Tamoxifen or Arimidex. These therapies cause vasomotor side effect, like the hot flashes or night sweats. Normally hormonal therapy can be used to treat these symptoms, but women who have had certain kinds of breast cancer cannot take those hormones so other treatments are necessary. One standard treatment is the anti-depressant Effexor (venlafaxine – a serotonin/norepinephrine reuptake inhibitor).
There were 47 patients in the study followed for 12 weeks in one of two arms – either acupuncture or standard treatment with Effexor. The study showed that acupuncture was as effective as Effexor but had fewer side effects and lasted longer after treatments stopped.
The press release also reports:
Findings also show there were additional benefits to acupuncture treatment for breast cancer patients, such as an increased sense of well being, more energy, and in some cases, a higher sex drive, that were not experienced in those patients who underwent drug treatment for their hot flashes.
Evaluating Clinical Trials
Readers of this blog will know that not all clinical trials are created equal, and there are some basic assessments that should be made in order to properly weigh the results of a study.
This study has many weaknesses – enough to make the results uninterpretable, which is jargon for “worthless.”
The fatal problem with this study is that it is not blinded – there is no sham acupuncture. Patients, therefore, knew if they received the novel treatment that was being studied. The long list of non-specific “benefits” perceived by the subjects who received acupuncture (such as an increased sense of well-being) is evidence for the placebo effect in this group. This weakness is enough to condemn the study.
Further, the study is small, with only 47 subjects. This is especially a problem for an equivalence outcome – the study showed no difference in degree of benefit between acupuncture and Effexor (although there was a difference in duration). This could simply mean that the study was not powerful enough to see any difference.
This small study size is also exacerbated by the fact that there was no placebo group – acupuncture was being compared to Effexor. It is therefore possible that in this study neither treatment worked – since we do not have a placebo control to gauge the absolute effectiveness of either treatment.
Prior studies indicate that Effexor is only “moderately” effective in treating hot flashes. So it is not an unreasonable control treatment, but the modest effectiveness of Effexor as a treatment makes the small sample size even more problematic.
Also, Effexor does have side effects because it is biologically active. It may be that Effexor has moderate benefit for hot flashes but these benefits are offset by the side effects in some cases. Meanwhile, acupuncture may have no effects or side effects -and so the net effect may have been similar.
The Disconnect Between Science and Press Release
Such a study as this is best described as a pilot study – meaning a small preliminary study. I have nothing against pilot studies – they serve an essential purpose in the arc of clinical research. They are useful for deciding if a clinical approach is worthy of further more definitive study. They should not be used as a basis for clinical decisions, however. They certainly should not be used to conclude that a highly implausible treatment has biological function.
There are two problems with this study, however. The first is the press release, which states:
“Our study shows that physicians and patients have an additional therapy for something that affects the majority of breast cancer survivors and actually has benefits, as opposed to more side effects. The effect is more durable than a drug commonly used to treat these vasomotor symptoms and, ultimately, is more cost-effective for insurance companies,” Eleanor Walker, M.D., lead author of the study and a radiation oncologist at the Henry Ford Hospital Department of Radiation Oncology in Detroit, said.
I find these statements to be completely misleading (to the point of being irresponsible) and not justified in any way by this study. It would only have been appropriate to say that this study shows acupuncture “may” be of use but requires further study – that’s what pilot studies are for.
The media, however, picked up the press release without criticism or meaningful analysis.
The Arc of Clinical Trials
My second problem with this study is to question the purpose of doing pilot studies at this stage of acupuncture research. Clinical research has a typical progress. It begins with observations, usually published as case reports or case series, suggesting a possible new connection or treatment. This then progresses to pilot studies (like this one) which are small and may be blinded or open.
If pilot studies are promising, or at least rule out any major risk or downside to a treatment, then larger studies are done that are designed to be definitive. However, this usually takes a few rounds, with protocols being refined in response to criticism or alternate interpretations of the data. Eventually, truly definitive or consensus trials are done and the clinical question is largely settled. (New data, however, can always revive an old question.)
Acupuncture has been studied for years, and trial design has evolved to well-designed trials that are double-blind and controlled by either sham acupuncture (where needles are inserted, but in the “wrong” place or not to depth) or, even better, placebo acupuncture where opaque sheaths hide from the practitioner and the patient whether or not a needle is inserted at all (the pressure from the sheath can disguise the sensation of the thin acupuncture needle being inserted).
At this point there is no reason to do pilot studies for acupuncture – this represents a regression of the typical research process. The only research that is of value at this point are well-blinded trials. However, such trials have failed to demonstrate that acupuncture has any specific effect (anything beyond placebo effects or perhaps non-specific effects from the process of getting acupuncture, such as relaxation).
It is reasonable to speculate that proponents of acupuncture are shying away from well-controlled trials because they have not given the positive results they desire. If this is not true, and pilot studies are being performed to explore the potential of specific applications of acupuncture, then the statements of these researchers in their press releases and to the public should reflect that – but they don’t. Rather, poorly designed or preliminary studies (which at this point are scientifically worthless) are used to make glowing endorsements of the effectiveness of acupuncture to the public.
This is not an isolated case. Three weeks ago a similarly worthless study of acupuncture for the treatment of headaches was published. This study, too, was not blinded and yet was used to promote the alleged effectiveness of acupuncture.
Acupuncture remains a highly implausible treatment. For this reason the threshold for evidence should be higher than for more plausible treatments. Acupuncture has also been studied enough for the technology of acupuncture research to have matured to fairly definitive studies, and to conclude that there is probably no large or easily detectable biological effect from acupuncture.
In addition, the popularity of acupuncture (while still small in absolute numbers) far outstrips the evidence for its effectiveness, making acupuncture a controversial treatment. This controversy is exacerbated by the existence of dedicated practitioners (acupuncturists), who have a vested interest in this one modality.
For these reasons any further testing of acupuncture should aspire to the highest scientific standards. Pilot studies of acupuncture are worse than worthless – they do nothing to further the scientific question, and they are abused to promote a dubious treatment through the credulous media.