I often get e-mail suggesting topics to cover on SBM or elsewhere in my social media content. I like getting these e-mails when they are organic, coming from readers here with genuine questions about some questionable claim or practice. But often they are press contacts, by a professional promoter pushing a new study or shopping around an author or someone they represent. These are rarely useful, but often ironically humorous, trying to promote some type of pseudoscience.
Recently I was sent a promotion for a study of Reiki, stating, “A new U.S. clinical study conducted by researchers from the University of Utah, Harvard and Florida State University has found that Reiki, a non-pharmacological, non-invasive therapy, significantly reduced symptoms of chronic knee osteoarthritis (KOA), supporting the feasibility of complementary therapies as drug-free pain management interventions.”
I have to say, I knew right away what the flaws in this study were likely to be, and I doubted the claim that the study properly shows that Reiki “significantly reduced symptoms”. Of course, “significant” can mean clinically significant or statistically significant, and for a clinical study to be meaningful you really need both simultaneously.
I doubted the claim being made for two reasons. The first is that there is zero plausibility for Reiki having any treatment-specific effects (other than nonspecific placebo-type effects). According to Reiki.org, Reiki “involves a practitioner, trained by a Master, placing their hands lightly on or just above a fully clothed recipient to guide “life force energy” (ki), aiming to clear energy blockages and support the body’s natural healing abilities.” Life force energy does not exist (as much as it is possible to say scientifically that anything does not exist), and therefore any claim to manipulate this non-existent force has no plausibility. Practitioners believe it is their intention to heal their client, which is what unblocks their life force energy, a phenomenon (affecting reality through pure intention) which also almost certainly does not exist. So using a non-existent method to affect a non-existent energy is, to say the least, an extraordinary claim.
Further, whenever I receive a claim that a study shows that some impossible thing is true, there is always a major flaw in the study. Always. By this I mean that the study does not justify rejecting the null hypothesis. They don’t even meet the threshold for doing so with a reasonably plausible hypothesis, let alone an extremely implausible one. So let’s dive into this study to show you what I mean.
The study is Complementary therapies for chronic knee pain: A placebo-controlled RCT of Reiki and mindfulness. There were four arms to this study – Reiki, placebo reiki which they call feiki (that name is the best thing about the study), mindfulness, and waitlist control. Right there I would have bet real money what the outcome of the study was – Reiki was significantly better than waitlist control, but not than feiki. And sure enough, that was the outcome. This is a negative study, being deceptively presented as a positive study. But let’s go into more detail.
The study is of symptoms from chronic knee pain, and the main outcome measure is a self-report of symptoms at baseline, one month, and two months. We have a study with a subjective outcome (nothing objective is being measured), which means that the control and blinding have to be exquisite or the results are not meaningful. The authors claim the study had “successful blinding.” However, this claim is based entirely on their attempts to blind the study, and not any measure of how successful the blinding was (such as asking participants if they thought they were in the treatment or placebo arm).
There is also the potential for asymmetrical study adherence. Even before allocation, however, of the 606 eligible subjects, 335 refused to participate. This suggests the possibility that subjects who agreed to the study were predisposed to believe in Reiki or mindfulness. After allocation – “Of the 132 participants that began treatment, 127 (96 %) were treatment completers. All but three participants completed Feiki, all but one completed Reiki, and all participants completed mindfulness training. Of the participants that began treatment, 89 % completed their post-assessment and 92 % completed the 2-month follow-up assessment.” So while there was a small dropout (still, higher in the feiki group), about 10% failed to complete their assessments.
Worse, despite claiming that the blinding was successful, there is good reason to believe that it was not. The big problem is that the people giving Reiki and feiki were not blinded. They were instructed to follow a script to minimize differences in interaction, but this was likely not enough. This is critical, given that the alternative hypothesis to a specific effect from Reiki, and the one that I think is supported by the evidence, is that the subjective effects are entirely due to the therapeutic interaction with the practitioner. It’s hard to imagine that these interactions were similar given that, “Feiki providers were trained to occupy their mind by counting backwards from 1000 by 7 s in their head while administering treatment.” This was to make sure they did not accidentally give “real” Reiki by intending to heal their subject.
So we have a subjective outcome with unblinded providers and some asymmetry in completion and a significant dropout of post-treatment evaluations, in a study with a modest size. This is hardly a compelling study. But arguably – none of this really matters, because the study was negative anyway. Here are the results:
“At the 2-month follow-up, participants in both the Reiki (x̄=32.64, 95 % CI=26.85–38.43, t = 2.29, p = .023, Cohen’s d=0.52) and Mindfulness (x̄=27.62, 95 % CI=21.94–33.29, t = 3.54, p < .001, d=.80) conditions reported significantly lower osteoarthritis symptom severity compared to those in the WLC condition (x̄=42.07, 95 % CI=36.40–47.75). Additionally, Mindfulness participants reported significantly less symptom severity than those in the Feiki condition (x̄=39.45, 95 % CI=33.76–45.14, t = 2.90, p = .004, d=.65. The difference between the Reiki and Feiki conditions approached statistical significance (t = 1.65, p = .10, d=.37), while symptom severity did not differ significantly between the Reiki and Mindfulness conditions (t = 1.22, p = .22, d=.28).”
“Approached statistical significance” is a euphemism for “not statistically significant”, i.e. negative. But the researchers try to rescue their results by doing an unusual analysis. They examined the “trajectories of change” and found that, between the 1-month and 2-month treatments, the mindfulness and Reiki groups improved, while the feiki group declined, and the slopes of these changes were significant. Did they really plan on making this very unusual analysis prior to the study? Did they do any other atypical analyses? Did they statistically adjust for making multiple comparisons? This reeks of p-hacking, and in any case does not change the fact that the primary comparison, and the only one that really matters in this study – a direct comparison of Reiki to feiki, was negative.
The study authors do their best to spin these negative results to make them seem encouraging. They do call their results “preliminary”, which is accurate, and then state, “While additional research is needed to confirm efficacy and elucidate underlying mechanisms, study findings contribute to the growing literature on Reiki and other Biofield Therapies as possible means of chronic pain management.”
Ah, the “additional research is needed” gambit. Sure, this is a legitimate way to characterize preliminary evidence. It is also a telltale sign of the limits and flaws in the evidence-based medicine (EBM) model, which fails to consider prior plausibility. I really don’t think additional research is needed. This was a negative study of a highly implausible (nigh impossible) hypothesis. Even with some wiggle room provided by questionable blinding (unblinded treatment providers with a highly subjective outcome) they couldn’t achieve statistical significance.
What happens with treatments like Reiki and similar low-probability treatments like acupuncture, which are essentially theatrical placebos, is that we are stuck in an endless cycle of preliminary evidence with significant flaws, and with the most rigorous comparisons being largely negative. Meanwhile proponents tout these ultimately negative and/or highly unconvincing results as showing these are viable alternatives to real medicine.
But we never ever get to a reasonable threshold for rejecting the null hypothesis. We never convincingly demonstrate that these treatments work or that any of the underlying principles are valid. Life force energy still does not exist, and you cannot heal patients with your intention.
