Perhaps one of the greatest threats to the enterprise of Science-Based Medicine is research fraud and misconduct. Rigorous research methods can be used to minimize the effects of bias, but when those methods themselves are the problem there is no easy fix. Related to this is the need for transparency. When fraud or misconduct is uncovered it erodes confidence in the system because it provokes speculation about how much fraud and misconduct has not been uncovered.
A recent study published in JAMA looks at one aspect of this issue – reporting of misconduct uncovered by the FDA. The good news here is that FDA trials, those that will be used to apply to the FDA for approval of a drug, are carefully monitored and inspected by the FDA. This is an important quality control measure. When the FDA uncovers misconduct it takes steps to correct it. If the misconduct is severe enough then any data that is associated with the poor research practices will be excluded from the trial so as not to taint the results. Even an entire study can be disqualified if necessary.
The problem highlighted by the study is that there is no systematic way for the FDA to communicate its findings through the peer-reviewed literature. Tainted studies, or ones that require a correction or retraction (because the violations were discovered after publication) may therefore persist in the peer-reviewed literature without any indication of the uncovered misconduct.
The sole author, Charles Seife, found:
Fifty-seven published clinical trials were identified for which an FDA inspection of a trial site had found significant evidence of 1 or more of the following problems: falsification or submission of false information, 22 trials (39%); problems with adverse events reporting, 14 trials (25%); protocol violations, 42 trials (74%); inadequate or inaccurate recordkeeping, 35 trials (61%); failure to protect the safety of patients and/or issues with oversight or informed consent, 30 trials (53%); and violations not otherwise categorized, 20 trials (35%). Only 3 of the 78 publications (4%) that resulted from trials in which the FDA found significant violations mentioned the objectionable conditions or practices found during the inspection. No corrections, retractions, expressions of concern, or other comments acknowledging the key issues identified by the inspection were subsequently published.
Seife gives some specific examples, includes an entire trial that was deemed unreliable by the FDA (the RECORD4 trial looking at an anticoagulant for deep vein thrombosis), but the FDA’s findings are not mentioned in any of the publications describing the trial’s results.
In a separate case Seife reports:
A clinical site in China taking part in a large trial of apixaban, a novel anticoagulant, had apparently altered patient records. If one were to exclude the data from the patients at that site, the claim of a statistically significant mortality benefit disappears.179 For this reason, among others, the FDA wrestled with whether it was appropriate to allow the manufacturer to claim a mortality benefit. None of this discussion appears in the literature.
Reporting of these findings exposes a serious concern, but one that has some obvious solutions. As Seife himself suggests, when the FDA finds violations it can be required to report their findings on clinicaltrials.gov. Seife also proposes that a separate database be created just to report FDA inspections that find misconduct. Further, peer-reviewed journals can require that submitting authors disclose any FDA violations, similar to the required reporting of potential conflicts of interest.
The FDA has responded to the study by saying they are:
…committed to increasing the transparency of compliance and enforcement activities with the goal of enhancing the public’s understanding of the FDA’s decisions, promoting the accountability of the FDA, and fostering an understanding among regulated industry about the need for consistently safe and high-quality products.
Regarding redaction of warning letters, the majority of information that is redacted from warning letters about clinical trials pertains to either the protocol or patient identifiers. The FDA redacts protocol information from warning letters because the information pertains to a pending application, not because it may reveal investigator misconduct. Information on pending applications is exempt from public disclosure based on several federal statutes and regulations, such as the Freedom of Information Act.
Mainstream reporting of these results have been, in my opinion, a bit hyped. This stems partly from the fact that Seife, who is an investigative journalist interested in exposing research fraud, wrote an editorial in Slate to accompany the publication of his study. Seife claims that the FDA is “burying” and has “hidden” evidence of misconduct. This is why the FDA addressed the issue of redacted warning letters in the quote above – redaction was interpreted as an effort to hide details of misconduct.
To put the problem into perspective, in 2013 FDA inspections uncovered misconduct in 2% of their investigations. Further, the FDA takes steps to correct the problems they uncover, such as excluding tainted data or even entire studies. The problem here is one of transparency, especially in the peer-reviewed literature. I don’t know if it is fair to say that the FDA buries its findings of misconduct, rather than they simply have no systematic method of reporting their findings.
As can be expected, the results of this study and the perhaps overly-hyped reporting have fueled conspiracy theories and science denial. Taken out of context, the study can be used to dismiss any scientific findings that one finds inconvenient to their ideology.
This is part of a broader problem, and one that we confront frequently at SBM. The institutions of science are certainly flawed, as is every human endeavor. Those flaws, however, are not fatal. The culture of science is also one that is self-reflective and contains a sincere motive for self-improvement and quality control.
We frequently point out flaws in the practice of science as a way to inform interpretation of the scientific literature, and to suggest specific improvements to make the practice of science more reliable and transparent. Others, however, point to the same flaws as a way of dismissing the findings of science in general, or at least anything with which they disagree.
Findings such as these can be used as a way of improving the institutions of science, or as a way of tearing it down. I prefer the former.