A recently published study claims to have shown that a proprietary mixture of velvet bean and Chlorophytum borivilianum improves sleep quality. The journal, Integrative Medicine Insights, is online, peer-reviewed, PubMed indexed, open-access, and it charges authors $1848.00 to publish their article. It advertises editorial decisions in 3 weeks and publication in 2 weeks after acceptance. I can see two reasons why authors might be willing to pay that much for publication: to speed the process of getting important research results out to the public, or because their research is poor quality and they know it would be rejected by other journals.
The quality of this study is unfortunately typical of much of the research on alternative medicine.
Description of Study
The full text is available for download here. The title is “A Dietary Supplement Containing Chlorophytum Borivilianum and Velvet Bean Improves Sleep Quality in Men and Women.” They gave a proprietary supplement mixture to 18 young healthy subjects with self-reported impairment of sleep quality (defined as routine difficulty falling asleep, waking more than twice during the night, and awaking in the morning feeling tired) and had them fill out a questionnaire about sleep quality before and after the trial. They also measured heart rate, blood pressure, CBC, metabolic panel, and lipid panels.
Subjects were told there was a 50% chance that they would get a placebo, but no placebos were actually used; this was a clumsy, inadequate attempt to minimize any placebo effect, and they didn’t even think to ask subjects afterwards whether they thought they’d received a placebo.
They described the subjects’ characteristics: average age, height, weight, waist and hip measurements, % body fat, years anaerobic exercise training, hours per week aerobic exercise, etc. The relevance of some of these characteristics is far from clear, and the information is next to useless because it is presented in a table that lumps everyone together and only provides average values +/- standard error.
The study lasted 28 days. Subjects were told to take 3 capsules a day for 3 days and then take between 1 and 3 a day depending on their tolerance and preference, in an attempt to mimic real life use of dietary supplements. They asked subjects not to change their diet; they had them keep food diaries and did detailed calculations of everything from calories to selenium(!?) based only on self-reported intake. Subjects were told to continue their usual exercise but to abstain for 24 hours prior to the days when tests were administered.
Results
On the Pittsburgh Sleep Quality Index (PSQI), a self-reported questionnaire, they found statistically significant improvement in subjective sleep quality, sleep latency, sleep duration, habitual sleep efficiency, sleep disturbances, daytime dysfunction, and PSQI global score. They found no significant adverse changes in vital signs or blood tests.
Bausell’s Checklist: Fail!
In his book Snake Oil Science, Bausell offers a simple 4-item checklist of things to look for to judge whether a study is likely to be credible:
- Randomized with a credible control group
- At least 50 subjects per group
- Dropout rate 25% or less
- Published in a high-quality, prestigious, peer-reviewed journal
There were no dropouts in this small study, but otherwise it fails.
Discussion Section
They hadn’t done a credible controlled study to prove that it did work, but that didn’t stop them from trying to explain how it worked. They speculated that the results were due mainly to the velvet bean component. Their rationale? Velvet bean seeds contain L-DOPA, L-DOPA has been shown to stimulate growth hormone (GH), and GH is “closely related to slow wave sleep.” Kind of convoluted, don’t you think? They cite another study that reported a significant increase in GH output in the 2 hours following ingestion of the supplement; but in the present study GH was not measured. Anyway, I don’t think there’s any evidence that patients sleep better when they take pharmacologic doses of GH for medical indications. They also speculated that the Chlorophytum component might have contributed to better sleep because of its saponin content and purported aphrodisiac qualities. The latter is a titillating conjecture, but aren’t aphrodisiacs supposed to excite people instead of putting them to sleep?
They knew they had done a flawed study. They said,
While these findings may be of interest, additional placebo-controlled studies are needed to corroborate these data, possibly including a larger sample size, increased length of treatment with the supplement, and additional time points of measurement. Additionally, further study with each ingredient independently would allow for an understanding of the influence of each on sleep quality.
Grammar Police Grade: B Minus
It always bothers me when a published study contains language errors that should have been noticed and corrected by editors, peer-reviewers, or proofreaders. It doesn’t necessarily mean that the study itself is a poor one, but it smacks of carelessness; and carelessness in language is often associated with carelessness in scientific method and reasoning. This article had a number of things I would have red-penciled, for instance “The PSQI is self-rated questionnaire” and calling blood tests “bloodborne variables.” Even the title is defective: the borivilianum in Chlorophytum borivilianum should not have been capitalized. Nitpicky, perhaps; but I never see errors like that in prestigious journals like The New England Journal of Medicine.
Other Considerations
I wondered about a couple of points that they failed to address. If the supplement only works indirectly by raising GH levels, wouldn’t it be more efficient to administer GH itself? Would that work? Even if it did, isn’t raising GH levels worrisome? GH is not innocuous: it is a hormone with multiple effects on human physiology. Would it be wise to raise it just to treat a non-serious condition like mild insomnia? And if velvet beans contain L-DOPA, what are the risks of treating otherwise healthy insomniacs with this powerful anti-Parkinson’s drug? Does it act as a sleeping pill when Parkinson’s patients take it in much higher doses? I don’t think so; in fact, one of the listed side effects of L-DOPA is insomnia.
Why Didn’t They Do a Better Study?
They had funding, they are trained scientists, they even recognized and explained some of the limitations of their study. Why didn’t they do a better study? Why didn’t they at least use a placebo control? Why did they confuse the issue by testing a product with two ingredients? Why so few subjects? Couldn’t they have eliminated some of those blood tests and used the money they saved to enroll more subjects? Why didn’t they use some objective measurement of sleep quality? A sleep lab might have been impractical and expensive, but a simple device like the Zeo might have provided more objective information than a sleep questionnaire. Self-reporting about sleep is notoriously unreliable (your grandfather’s complaint that he “didn’t sleep a wink” is belied by the loud snoring you overheard). Their results might seem to support the hypothesis that the supplement is effective, but they are just as compatible with the hypothesis that any intervention will result in self-reported improvements in sleep due to suggestion and expectation. They may have simply provided yet one more demonstration of the placebo effect.
Why do people do studies like this? Why do editors publish them? Why aren’t peer reviewers more critical? When you consider the limited funds available for research, this kind of thing is a real tragedy. Why not throw out 10 or 20 poor-quality research proposals and use the funds for one well-designed study whose results might actually mean something?