[Editor’s note: Dr. Gorski decided to take the Fourth of July off and will return next week. Thankfully, Dr. Howard was happy to step in with this post.]
Dr. Peter Doshi, an associate professor of pharmaceutical health services research, is familiar to readers of SBM as someone who starts with the premise that vaccines are unsafe and ineffective, and then works backwards to find the evidence. His most recent attempt (along with Joseph Fraiman and other coauthors) to torture data to support his preconceived conclusions was a preprint titled “Serious Adverse Events of Special Interest Following mRNA Vaccination in Randomized Trials“. This paper analyzed both the Pfizer and Moderna RCTs, and predictably concluded:
The excess risk of serious adverse events of special interest surpassed the risk reduction for COVID-19 hospitalization relative to the placebo group in both Pfizer and Moderna trials.
That sounds bad, and sheltered virus-advocates were happy to spread this message far and wide (here and here).
At first glance, simply tallying the harms suffered by those in the vaccine and placebo groups sounds like a reasonable approach. But is it? Well, it all depends on how those harms are counted and for how long they are counted.
Dr. Susan Oliver, a scientist who studies nanomedicine at The University of New South Wales, made a short, must-watch video exposing the statistically chicanery Dr. Doshi used to reach his conclusions. Dr. David Gorski also discussed Dr. Doshi’s tricks here. Both Drs. Oliver and Gorski noticed that in determining “how these harms are counted”, Dr. Doshi ensured that the potential harms of the vaccine were inflated relative to harms of the virus. A single person who had gastroenteritis and abdominal pain after vaccination was counted as having two adverse events, while someone who was hospitalized with COVID, was counted once, even if they experienced a multitude of grave complications as a result.
Beyond this, Drs. Oliver and Gorski identified an even bigger problem. They realized Dr. Doshi’s entire paper was an exercise in comparing apples to oranges. To understand why, let’s briefly review the key results of the original COVID vaccine RCTs from the end of 2020.
- In the Pfizer RCT, 21,720 people received the vaccine and 21,728 received placebo. There were 162 COVID cases in the placebo group and 8 in the vaccine group. There were 10 cases of severe COVID-19, 9 of which were in the placebo group, 1 the vaccine group.
- In the Moderna RCT, there were 15,210 participants in the vaccine and placebo groups. Symptomatic COVID occurred in 185 participants in the placebo group and in 11 participants in the vaccine group. Severe COVID occurred in 30 participants, with one fatality, all in the placebo group.
We must now acknowledge two key points. First, depending on how quickly a virus spreads, the benefits of a vaccine can take many months, even decades to accumulate. Second, nearly all vaccine harms occur right after vaccination. Someone who analyzed the RCTs after one week would find the vaccine caused a bunch of people to feel unwell, while preventing zero cases of COVID. In all vaccine RCTs, the harms are frontloaded, while the benefits accrue gradually over time. Just look at those famous graphs from the COVID RCTs to see how obvious this is. Every month, the benefits from the vaccine became larger and larger – and the trial only lasted several months. In contrast, I can confidently state that few new vaccine harms were being reported at the trial’s end.
As a result, to adequately capture the vaccine’s benefits, an RCT would have to last the entire pandemic. As more and more people were exposed to the virus, the vaccine’s benefits relative to placebo would grow and grow, even considering that vaccine efficacy wanes. Had the trial lasted two years, there would be many more cases of severe COVID, especially in the unvaccinated cohort. As Dr. Oliver pointed out, an alternative to a pandemic-long RCT would be a challenge trial. Purposefully exposing participants to the virus would also create the apples to apples comparison Dr. Doshi pretends he has.
The raw numbers from the RCTs expose another problem with Dr. Doshi’s paper. The number of participants who received the vaccine was much larger than the number of participants who were exposed to the virus. In these two trials, there were 74,000 participants, 36,930 of them received a vaccine, while only 347 of them had COVID. The vaccine had many more opportunities than the virus to make people feel rotten during the RCT.
This was no accident. The RCTs were designed to end when a certain number of people contracted COVID. According to the Moderna protocol, “The primary analysis will be performed when approximately 151 cases have been observed in the study”. The Pfizer protocol set “a target of 164 primary-endpoint cases of confirmed Covid-19”. So by design, the trials concluded before the virus was allowed to harm more than several hundred people.
Notably, nearly all of those harmed by the virus received placebo. Like all subsequent studies, the RCTs showed that COVID was dangerous and the vaccine very effective at curtailing it’s dangers. There were 40 cases of severe COVID, all but one in the placebo cohort. That’s a pretty big deal considering that many more of those 74,000 trial participants have contracted COVID by now. Once this protection against severe COVID was known, it would have been unethical to continue the trials and allow people to remain unvaccinated. The trials had to end there.
When Dr. Doshi claims his “results show an excess risk of serious adverse events of special interest greater than the reduction in COVID-19 hospitalizations in both Pfizer and Moderna trials”, he doesn’t acknowledge these studies were terminated precisely because those hospitalizations began to accumulate. The trials were designed to stop once a small number of people got COVID, and Dr. Doshi spins this to claim COVID wasn’t much of a threat. These trials didn’t prove COVID wasn’t a threat. Rather they ended so rapidly precisely because COVID was a threat. Dr. Doshi and his many co-authors have cause and effect completely backwards.
Their “finding” was an entirely unsurprising, if not inevitable outcome from the way these trials were designed. There were never going to be more than several hundred people to be harmed by COVID in these studies. Dr. Doshi’s study would be of value only if the virus ceased to be a threat or the vaccine ceased to be of any benefit as soon as the RCT ended. Of course, even this fantasy would require that he tally the harms in a balanced way.
We all know what happened once the RCTs ended, and this has great bearing on Dr. Doshi’s preconceived conclusions. First, the virus spread everywhere. In the RCTs, there were 100 vaccinated people for every person who had COVID. That ratio is very different now, and this renders Dr. Doshi’s method of counting harms completely obsolete. Second, worse variants arrived, and these too nullify Dr. Doshi’s tallying method. If these RCTs were done when Delta was circulating, there would have been more than 40 cases of severe COVID, mostly in the unvaccinated group. This is because while vaccine efficacy against symptomatic infection drastically waned, the vaccine retained much of its benefit at preventing severe outcomes, especially when augmented by a third dose.
This technique of minimizing vaccine benefits by only focusing on RCTs of relatively short duration is well-known to vaccine-advocates. In my previous article on methodolatry, the inappropriate worship of RCTs alone, I described how anti-vaxxers are fond of pointing out that the HPV vaccine has never been shown to prevent cancer in an RCT. They’re right. But it was never a great leap to speculate that a vaccine that prevented HPV infection and pre-cancerous lesions in an RCT would eventually be shown to prevent HPV-related cancers. Indeed, there is now overwhelming evidence from observational studies this is the case.
Given this, honest brokers do not limit themselves to RCTs when evaluating vaccines. While observational studies are subject to more bias and cannot conclusively determine causality, unlike most RCTs, they can evaluate large numbers of people for long periods of time. As a result, they can reveal many things that RCTs can’t. We’ve learned much more about these vaccines from observational studies and billions of doses over two years than from the RCTs, where only 36,930 people received a vaccine, and only 366 people contracted COVID.
Anti-vaxxers are perfectly content to promote observational data, so long as it purports to reveal vaccines’ flaws. Notably, Dr. Doshi cited non-RCT data, including a VAERS dumpster dive, to highlight rare vaccine side-effects. In contrast, observational data is unmentionable to anti-vaxxers when it shows vaccine are safe and effective. Dr. Doshi neglected to reference a single non-RCT showing the vaccine’s benefits in his paper.
There is no shortage of such studies. The evidence is overwhelming that COVID vaccines are very safe and they keep people alive and out of the hospital. Only someone who starts with the conclusion that vaccines don’t work and then works backwards to find the evidence could claim otherwise.