While scanning through recent science press releases I came across an interesting study looking at the use of a pharmaceutical grade antioxidant, N-Acetylcysteine (NAC), in the treatment of certain symptoms of autism. This is a small pilot study, but it did have a double-blind placebo controlled design. The press release reports:
During the 12-week trial, NAC treatment decreased irritability scores from 13.1 to 7.2 on the Aberrant Behavior Checklist, a widely used clinical scale for assessing irritability. The change is not as large as that seen in children taking antipsychotics. “But this is still a potentially valuable tool to have before jumping on these big guns,” Hardan said.
But concluded:
“This was a pilot study,” Hardan said. “Final conclusions cannot be made before we do a larger trial.”
I also noticed that two of the authors list significant conflicts of interest – patents on the use of NAC, and one has equity in the company that makes it. It occurred to me that a larger question than the efficacy of NAC for these autism symptoms is this – if this is a pilot study only and we should not base any firm conclusions on the results, then why the press release?
In a broader context, we are currently going through an information revolution. There are many positive aspects to the easy access to information provided by the internet, but it is a disruptive technology in many ways. The news industry is desperately trying to adapt to the collapse of their traditional business model, for example. Relevant to my question above, the scientific community is now exposed to public scrutiny in a dramatic way that is very different from the pre-internet age.
It used to be that studies were published in print scientific journals that would largely be read by experts in the field. Journal articles and presentations at scientific meetings were (and continue to be) an important conversation that each scientific community has with itself, to review new findings and ideas. This conversation happened largely in isolation from public view, because it really is only interesting and relevant to researchers. Most new ideas in science are not going to pan out. The meat grinder of peer-review, replication, and analysis by the scientific community weeds out many hypotheses (which in the field of medicine are potential new treatments).
As new hypotheses survive through the process of peer-review and further research, the valuable ideas will tend to survive while the rest fade away (at least ideally that is what should happen). At some point scientific ideas become established enough that they begin to filter their way into the science section of newspapers, articles in popular scientific magazines, and then finally into documentaries and text books. I won’t pretend this system ever worked perfectly, but at least there was a relationship between the degree to which scientific ideas were validated and their penetration into public awareness. Anyone really interested in the latest research could read primary journals or attend meetings, but preliminary findings were not often presented as news to the public.
The internet has changed this. Now scientific journals often publish their abstracts online, and there are some journals that have full public online access. On the whole I think this is a very good thing. It facilitates research and the exchange of information. I certainly cannot imagine maintaining an information source like Science-Based Medicine, with almost daily articles, without instant online access to published articles.
The scientific community has to realize, however, that as a consequence of this the conversation that scientists used to largely have with each other is now happening more in the public eye. In medicine this means that patients are reading preliminary studies and making health care decisions based upon the findings.
The issue within medicine is similar to the broader issue created by the democratization of information by the internet – the explosion of access (both on the creation and consumption end) increases freedom but decreases quality control (which in medicine we call the standard of care). Journalists argue that a million amateur bloggers are not a replacement for even a single full time, trained, investigative journalist, and they have a point.
I am not arguing for limiting access to information. That genie is out of the bottle, and I think the positive effects of information access far outweigh the negatives. But I do think that the scientific and medical communities need to recognize the new reality they are living in and adapt to it. I have some specific recommendations.
The first is not to send out press releases for preliminary pilot studies. The studies are there for those who are interested to find, but sending out a press release almost guarantees the results will be published as news, and the caveat at the very end about the results being preliminary is likely to be lost on most of the public. Essentially the authors are saying, here are some interesting results, now don’t make any health decisions based upon them. It’s not only unrealistic, it’s irresponsible. Over reporting of preliminary results, most of which are likely to be wrong or misleading, is a disservice to the public. The noise it generates will lead to confusion, and may lead to a decrease in public confidence in the scientific community.
In the case of NAS and autism, the authors acknowledge that NAS as a supplement is already being used by “alternative” practitioners to treat autism. Further, the NAS used in the study is pharmaceutical grade and specially handled to prevent break down of the chemical, and the results (in addition to being preliminary) cannot necessarily be applied to the supplements being sold. So this study is a setup to be misused to promote an existing market for an unproven therapy.
Journals should also consider more clearly labeling preliminary studies they publish, and perhaps even creating a special section in the journal for such studies. The reason for this is that the “peer-reviewed” label gets attached to definitive and preliminary research alike, and often this label is used to promote an idea to the public. If they were published in a section of the journal designated for preliminary findings, that might help reduce confusion over what peer-reviewed actually means. It would at least provide an easy alert to journalists and news outlets – this article comes with a huge warning at the top that the results are preliminary only (not a short caveat at the end).
I also believe that the scientific community needs to be more involved in reporting science news to the public. Press releases are often handled by non-scientists working in the press office of a journal or university, and often they distort or overhype the research. Their goal is not to accurately report results, but to promote their institution and grab headlines. The dwindling science journalism infrastructure means that many news outlets just pass along the press releases, without doing any actual journalism themselves. Often generalist reporters are reporting science news stories they are not equipped to understand. We need to rebuild the science journalism infrastructure by finding ways to support journalists who are trained to report science news, and training scientists to be their own journalists or at least how to deal with the media.
Finally we need to include in our science education curriculum how to skeptically read science news. The public needs to become more scientifically literate, because no matter what steps we take to improve how science news is reported to the public, there will always be a tremendous amount of bad or misleading science reporting (again – the internet genie is out of the bottle).
Conclusion
The scientific community needs to recognize that the world has changed around them. The conversation that scientists used to have mainly with each other is now increasingly happening in full public view. This can ultimately be a powerful boon to science, which needs to be transparent and open. The scientific community, however, has to adapt to this new reality. It has to retool the conversation to account for the public attention, and it has to engage more energetically with the public. Further it should become more directly involved in promoting higher quality science education in our society.
