Shares

I’ve been writing about how antivaxxers misuse the Vaccine Adverse Events Recording System (VAERS) database dating back to very early in my history as a blogger. VAERS, for those unfamiliar with it, is a database to which anyone can report any adverse event (AE) noted after vaccination. It serves as an early warning system that can generate hypotheses regarding correlations between specific AEs and vaccines, but by its very design cannot test these hypotheses. (I’ll explain in detail later in this post when I discuss the shortcomings of VAERS.) Indeed, out of curiosity, I did some searching of all of my blogs and discovered that the first time that I mentioned VAERS and its shortcomings was in 2005 in the context of a discussion of chelation therapy for autism. The first time that I ever discussed VAERS here on Science-Based Medicine (SBM) was in 2008 in the context of a discussion of harms attributed by antivaxxers to the human papilloma virus (HPV) vaccine. Because of its very nature, VAERS is prone to misunderstanding and misuse, and, unsurprisingly, VAERS has become a favorite tool of antivaxxers to claim that vaccines cause whatever AE on which they want to blame vaccines. That’s why, over the years, we’ve mentioned VAERS on this blog many times, and I’ve mentioned it elsewhere many times going back to 2005.

Last week, VAERS was in the news, and for once it wasn’t because of antivaxxers abusing it. Unfortunately, that makes the misuse of the database all the more potentially damaging. You see, there was a study published as a preprint last week, and it’s being held up as “evidence” that vaccinating 12-17 year olds against COVID-19 is more dangerous than COVID-19 because of post-vaccination myocarditis. It goes along with a common refrain that a lot of physicians who really should know better have been repeating without context and, sadly, even though they are not antivaccine need to be fact-checked as though they were, namely the false narrative that masks and vaccines to mitigate the spread of COVID-19 are at best useless in children or even worse than the disease.

In adding to this refrain, this study contributes material to the antimask and antivaccine movement and falls prey to the same sorts of gross errors observed in studies done by antivaxxers using VAERS data. You’ll note that guest blogger Dr. Dan Freedman has already commented on some of the glaring problems with this study. I view my commentary as complementary in that I will provide a lot of background that I’ve learned the hard way over nearly 20 years and emphasize different problems than the ones Dr. Freedman did. Moreover, I’ll cite problems pointed out by others on social media with more expertise than I in the nitty-gritty of these sorts of analyses, to complement my knowledge of VAERS. I will begin by describing how this study is being spun, continue to discuss the history and structure of VAERS, which is absolutely essential to understand why this study is so bad, and then finish by delving into the study itself, using the analysis to make some observations.

Prelude: Doctors who should know better stumbling through VAERS

As many times as I’ve seen antivaxxers misuse VAERS to try to blame vaccines for autism, sudden infant death syndrome, infertility, and more, I must admit that I was unprepared and profoundly disappointed late last week to see the database abused in a similar fashion by investigators who are not antivaccine (although definitely biased) and then to see their bad science amplified in stories like this one in The Guardian by its science editor Ian Sample (who really should know better), titled “Boys more at risk from Pfizer jab side-effect than Covid, suggests study“:

Healthy boys may be more likely to be admitted to hospital with a rare side-effect of the Pfizer/BioNTech Covid vaccine that causes inflammation of the heart than with Covid itself, US researchers claim.

Their analysis of medical data suggests that boys aged 12 to 15, with no underlying medical conditions, are four to six times more likely to be diagnosed with vaccine-related myocarditis than ending up in hospital with Covid over a four-month period.

Most children who experienced the rare side-effect had symptoms within days of the second shot of Pfizer/BioNTech vaccine, though a similar side-effect is seen with the Moderna jab. About 86% of the boys affected required some hospital care, the authors said.

This study actually suggested nothing of the sort, for reasons that I will explain. In fairness, though, let’s see what the study’s first lead author had to say last week when the study first hit the preprint server medRxiv:

And, of course, the obligatory “I’m not antivaccine; I’m pro-vaccine”:

No, I don’t think that Dr. Tracy Høeg (or any of the other authors) is antivaccine. However, it is very clear that she and her co-authors are completely out of their depth here and do not understand how VAERS works, even though Dr. Høeg, a sports and spine medicine specialist, does also have a PhD in Epidemiology and Public Health from University of Copenhagen. She really should know better, but this study demonstrates that she does not, as I will show. As for the rest of the authors, Allison Krug is also an epidemiologist who, again, really should know better, while Dr. John Mandrola is an adult cardiologist with a definite axe to grind who bills himself as a “medical conservative”. As for Josh Stevenson, I had never heard of him before this, but he is listed as being with a corporation called Truth in Data, LLC. I couldn’t find out much more about him other than that and what’s on his Twitter feed.

[ADDENDUM: Since this post went live earlier this morning, several people have reached out to inform me that Stevenson is a member of the team behind Rational Ground. I have therefore added this addendum, plus some additional Tweets below about this. A brief perusal of Rational Ground website reveals a group dedicated to minimizing the severity of COVID-19 and resisting mask mandates and lockdowns, and his Twitter feed is consistent with that. Interestingly, his Twitter bio does not list his affiliation with Rational Ground, almost as though he didn’t want anyone to know it. Indeed, a Google search of his name plus “COVID” doesn’t bring it up in the first three long pages of results. (Maybe he’s embarrassed, or maybe he doesn’t want people citing his work to be aware of his bias.) Interestingly, no one on the Rational Ground team appears to have anything resembling the necessary expertise to analyze COVID-19 data, including Stevenson, who describes himself as a “data visualization expert who focuses on creating easy to understand charts and dashboards with data” whose “background is in computer systems engineering & consulting, and his Bachelor’s degree is in Audio Engineering.” Quelle surprise.]

Unfortunately, even though it’s a preprint that has not yet been peer-reviewed, it’s been picked up by the usual suspects and amplified on social media. For example, Dr. Jay Bhattacharya was ebullient:

Bhattacharya clearly understands VAERS as well as the authors of this paper (i.e., not at all); otherwise, he would not have said anything quite so cringe-inducingly wrong. On the other hand, he is one of the main signatories of the Great Barrington Declaration, which advocated basically doing nothing about the pandemic other than “focused protection” of the “vulnerable,” apparently blissfully unaware that it’s impossible to protect the vulnerable if COVID-19 is spreading unchecked through the entire population.

Dr. Vinay Prasad was even more enthusiastic:

I would hardly call these authors a “dream team” (especially Josh Stevenson, who has no requisite expertise to help carry out such an analysis). I will, however, admit that, unlike Dr. Prasad’s previous characterization of combatting health misinformation as LeBron James “dunking on a 7′ hoop,” deconstructing this paper is a little harder, maybe like dunking on an 8′ hoop, simply because it requires a detailed explanation of how VAERS works and why it can’t be used to do what the authors used it for. I also can’t help but note that Dr. Prasad is well-known for demanding more rigorous data for various medical interventions, such as masking to slow the spread of COVID-19. Yet on this topic, he seems quite happy with a low-quality study that misuses the VAERS database, which is why this unrelated assertion made me laugh:

I can’t really comment on these indications for bone marrow transplantation, but Dr. Prasad is oddly oblivious to the weakness of the study by Høeg et al.

Moreover, as was pointed out over the weekend, the rapidity with which this preprint study made it into major news outlets of a certain ideological bent reeks of astroturfing:

Worst of all, though, politicians opposed to vaccine mandates and masking are using the study to justify their message:

Never mind that it is not yet even peer-reviewed. Bad science has real consequences:

Let’s dig in and explain what’s wrong with the study. Before I dig into the weeds of the study, its methodology, and its conclusions, though, a brief history of VAERS is necessary to provide context. If you’re already very familiar with VAERS, its proper uses, its weaknesses, and how it’s been misused by antivaxxers, you can probably skip the next section, although you might still at least want to skim it.

A brief history of VAERS and why it is not a reliable data source

VAERS is a bit of an odd beast when it comes to vaccine safety reporting systems. It was established jointly by the FDA and CDC in 1990 as an outgrowth of the National Childhood Vaccine Injury Act of 1986 and was intended as an “early warning system” or, as I like to call it, a “canary in the coalmine”. To accomplish that function, the CDC and FDA designed VAERS as an open system that allows anyone to enter any suspected AE after vaccination. It’s also an open system in that the data are freely available on the VAERS website to search for reports of AEs after vaccination. Anyone can do it. (I have.) Unfortunately, seemingly everyone has, whether they understand VAERS or not.

According to the CDC, the objectives of VAERS are to:

  • Detect new, unusual, or rare vaccine adverse events;
  • Monitor increases in known adverse events;
  • Identify potential patient risk factors for particular types of adverse events;
  • Assess the safety of newly licensed vaccines;
  • Determine and address possible reporting clusters (e.g., suspected localized [temporally or geographically] or product-/batch-/lot-specific adverse event reporting);
  • Recognize persistent safe-use problems and administration errors;
  • Provide a national safety monitoring system that extends to the entire general population for response to public health emergencies, such as a large-scale pandemic influenza vaccination program.

However, there are a lot of caveats, as you can see from the CDC’s own description of VAERS:

Established in 1990, the Vaccine Adverse Event Reporting System (VAERS) is a national early warning system to detect possible safety problems in U.S.-licensed vaccines. VAERS is co-managed by the Centers for Disease Control and Prevention (CDC) and the U.S. Food and Drug Administration (FDA). VAERS accepts and analyzes reports of adverse events (possible side effects) after a person has received a vaccination. Anyone can report an adverse event to VAERS. Healthcare professionals are required to report certain adverse events and vaccine manufacturers are required to report all adverse events that come to their attention.

VAERS is a passive reporting system, meaning it relies on individuals to send in reports of their experiences to CDC and FDA. VAERS is not designed to determine if a vaccine caused a health problem, but is especially useful for detecting unusual or unexpected patterns of adverse event reporting that might indicate a possible safety problem with a vaccine. This way, VAERS can provide CDC and FDA with valuable information that additional work and evaluation is necessary to further assess a possible safety concern.

Then there’s this disclaimer:

VAERS accepts reports of adverse events and reactions that occur following vaccination. Healthcare providers, vaccine manufacturers, and the public can submit reports to the system. While very important in monitoring vaccine safety, VAERS reports alone cannot be used to determine if a vaccine caused or contributed to an adverse event or illness. The reports may contain information that is incomplete, inaccurate, coincidental, or unverifiable. In large part, reports to VAERS are voluntary, which means they are subject to biases. This creates specific limitations on how the data can be used scientifically. Data from VAERS reports should always be interpreted with these limitations in mind.

The strengths of VAERS are that it is national in scope and can quickly provide an early warning of a safety problem with a vaccine. As part of CDC and FDA’s multi-system approach to post-licensure vaccine safety monitoring, VAERS is designed to rapidly detect unusual or unexpected patterns of adverse events, also known as “safety signals.” If a safety signal is found in VAERS, further studies can be done in safety systems such as the CDC’s Vaccine Safety Datalink (VSD) or the Clinical Immunization Safety Assessment (CISA) project. These systems do not have the same scientific limitations as VAERS, and can better assess health risks and possible connections between adverse events and a vaccine.

In brief, the key strength of the VAERS database is its open nature, which allows instantaneous reports from anyone, health provider or lay person, which in turn provides the data for rapid hypothesis generation. However, the key weakness of the VAERS database is also its open nature, which can lead to bias and the inclusion of incomplete, inaccurate, coincidental, and unverified information. Indeed, let’s go back to my first mention of VAERS in 2005, which I wrote after learning about a now-famous anecdote from James Laidler:

The chief problem with the VAERS data is that reports can be entered by anyone and are not routinely verified. To demonstrate this, a few years ago I entered a report that an influenza vaccine had turned me into The Hulk. The report was accepted and entered into the database.

Because the reported adverse event was so… unusual, a representative of VAERS contacted me. After a discussion of the VAERS database and its limitations, they asked for my permission to delete the record, which I granted. If I had not agreed, the record would be there still, showing that any claim can become part of the database, no matter how outrageous or improbable.

Since at least 1998 (and possibly earlier), a number of autism advocacy groups have, with all the best intentions, encouraged people to report their autistic children—or autistic children of relatives and friends—to VAERS as injuries from thimerosal-containing vaccines. This has irrevocably tainted the VAERS database with duplicate and spurious reports.

Laidler’s example is not the only one. In 2006, in response to a lot of commenters crying “BS!” after having read about Laidler’s anecdote, another autism advocate also known for combatting antivaccine misinformation then decided to see if he could replicate Laidler’s experience. He could:

VAERS has two ways of submitting a report. Firstly, you could download a PDF, fill it in and post it off. Or, you could do what I elected to do and fill in and submit a report online.

VAERS has a helpful popup which tells you exactly what it needs to know – which are the most important pieces of data it needs. However, the fact that I live in the UK was not deemed of importance. Neither was the fact that I told VAERS that my daughter had been turned into Wonder Woman. The only piece of contact data I submitted was my email address and I wasn’t even asked for that. I submitted it voluntarily.

As Jim Laidler himself once put it:

You see, the VAERS database is not a good source of material for any epidemiological study, a fact acknowledged by the very people who maintain it. The purpose of the VAERS database is to act as a repository for any complications following a vaccination. These complications may be due to the vaccines or may be coincidental, but the VAERS team wants to hear about them all. After all, their purpose is to watch for any unexpected consequences of vaccinations.

This is a statement that, I hope, answers the question that I’m sure many of you have: Why on earth would the CDC and FDA maintain such a flawed database in the first place? I also like to note that, whatever the merits or lack thereof in maintaining a database like VAERS, contrary to the way it is portrayed by antivaxxers, VAERS is by no means the be-all and end-all of vaccine safety monitoring. Antivaxxers love VAERS because its flaws make it easy to use to produce seeming correlations between vaccines and autism (and all the other things they blame vaccines for), but there are other systems that are active surveillance systems, such as the Vaccine Safety Datalink (VSD) and the Clinical Immunization Safety Assessment (CISA) project. These are two of the systems used to test the hypotheses generated by VAERS.

That same year, I discussed how attorneys suing vaccine manufacturers for “vaccine-induced” autism (never mind that vaccines do not cause autism) had gamed the VAERS database for litigation purposes by encouraging their clients to submit reports of autism as an AE after vaccination, thus hugely distorting the database. At the time, I noted how this feature of VAERS makes it inherently unreliable for as a tool for longitudinal studies of the rates of vaccine-related AEs. Regular readers will remember how I keep harping on “baseline rates” of AEs (the rate that would occur in the absence of vaccination). The problems with all studies of VAERS are that there is no “control group” and there is no way of knowing whether the sample in VAERS is a representative one. (In fact, it certainly is not.)

That’s why those of us who’ve written about VAERS like to refer to studies like this as “dumpster-diving” or “garbage in, garbage out”. True, I didn’t invent the term “dumpster diving” to describe these VAERS studies, but I have used it to describe various misuses of VAERS, starting with the father-son duo of Mark and David Geier’s dumpster dive for autism in 2006, followed by other examples like:

There are more examples, of course. Unfortunately, in the age of the COVID-19 pandemic, those of us familiar with this history of VAERS warned that antivaxxers would weaponize the database to blame all sorts of complications on COVID-19 vaccines. Nobody listened. Unfortunately, it has come to pass, as I’ve discussed multiple times on this blog, when antivaxxers have pointed to VAERS to try to blame the vaccine for a “holocaust” or even “depopulation” based on VAERS reports, something I documented as early as February, less than two months after the EUA for the Pfizer vaccine. Indeed, blaming the COVID-19 vaccines for thousands of deaths and a number of other complications based solely on their own analysis of VAERS data is a prominent feature of antivaccine arguments, to the point that Robert F. Kennedy, Jr. maintains a weekly update of fear-mongering VAERS reports.

Into this history boldly stride Høeg et al. It does not go well.

Dumpster-diving for myocarditis

So let’s circle back to the study (“SARS-CoV-2 mRNA Vaccination-Associated Myocarditis in Children Ages 12-17: A Stratified National Database Analysis“) by Høeg et al., who fall prey to the same sorts of missteps in analyzing VAERS as the Geiers, Seneff, Goldman, Miller, and a depressing array of antivaxxers during the 17 years I’ve been blogging and, I’m guessing, before. Things go very wrong right from the first sentence of the abstract that describes the objectives of the study as:

Establishing the rate of post-vaccination cardiac myocarditis in the 12-15 and 16-17-year-old population in the context of their COVID-19 hospitalization risk is critical for developing a vaccination recommendation framework that balances harms with benefits for this patient demographic.

And, near the end of the introduction:

Our primary aim was to stratify post-mRNA vaccination myocarditis by age and vaccination dose within the 12–17-year-old population. Our secondary aim was to provide an updated estimate to complement the CDC’s [2,3,6] and FDA’s [4] findings. Our final aim was to perform a harm-benefit analysis of mRNA COVID-19 vaccination myocarditis with that of COVID-19 hospitalization for children with and without one or more comorbidity at low, moderate, and high 120-day COVID-19 hospitalization rates.

I was half-tempted just to stop right here, throw my hands up, do a dramatic facepalm, and rant that you can’t do this with just VAERS data, much less compare your VAERs-estimated risk of myocarditis between age groups and then compare that risk to the risk of hospitalization from COVID-19 and leave it at that. (I know. You can do it, but you’ll end up with unreliable results, because by its very nature VAERS does not allow for an accurate estimate of the frequency or incidence of any given AE after vaccination using its public-facing data.) All VAERS can do is to detect potential safety signals that require other monitoring systems to determine if these signals are spurious or represent real correlation and causation by the vaccine. (After all, what have I just spent something like 2,500 words discussing in the context of the history and design of VAERS?) However, I suspect that the authors would become very offended and indignant and then argue that they can indeed do such a thing, pointing to their methods. That leaves me little choice but to go into the weeds of their methods, as noxious as those weeds are. Also, they will point out that the CDC itself has used VAERS data to estimate this frequency, which is true but incomplete and misleading.

That brings me to another point, just as important. Why on earth did these investigators basically repeat the analysis that the CDC and FDA did for a recent meeting of the Advisory Council on Immunization Practices (ACIP), only without all the extra data that ACIP has? As our guest blogger from earlier today put it:

Exactly. Unlike Høeg et al., ACIP had access to data adjudicated by investigators at VAERS and the CDC to determine if the AE actually happened and to assess the likelihood that it might have been due to the vaccine. ACIP also had access to up-to-date data from VSD, a far more reliable source. So I echo Dr. Freedman’s sentiment: What on earth did the authors hope to accomplish by, in essence, repeating that analysis, but with unadjudicated and much less complete data? Choosing to do this suggests an ideological bias, that the authors didn’t believe the CDC, FDA, and ACIP. I don’t expect everyone just to take the word of ACIP that its analysis is the be-all and end-all on this question, but a measure of humility is called for. As Dirty Harry once said, “A man’s got to know his limitations”, and the Høeg et al. clearly did not know their limitations or the limitations of the dataset they chose to use. That it was open, convenient, and easily accessible does not excuse them, particularly given that they could have requested the adjudicated dataset:

They could also have gotten access to VSD data. All that’s required is a written protocol approved by an institutional review board (IRB). They did not, which leads to these questions:

And:

Did the authors’ biases lead to this analysis? Who knows? I just know that the result is definitely not good, and the involvement of Josh Stevenson in particular makes this whole study very suspect indeed.

In any discussion of using VAERS data for anything, the search strategy is all, because in any epidemiological study case ascertainment is all. Høeg et al. make a big deal about aligning their inclusion criteria “with the CDC working case definition for probable myocarditis” and “same objective findings of cardiac injury used by the CDC to identify probable cases”, but their search strategy is rather broad:

We searched the Vaccine Adverse Event Reporting System (VAERS) data for females and males ages 12-17 in reports processed from 1/1/2021 through 6/18/2021 with diagnoses of “myocarditis,” “pericarditis,” “myopericarditis” or “chest pain” in the symptom notes and required the term “troponin” in the laboratory data. We defined a CAE using the CDC working case definition for a probable case.[2] Specifically, the symptom of “chest pain” required at least one of the following: diagnosis of myocarditis, peri- or myopericarditis, acute myocardial infarction; elevated troponin; abnormal electrocardiogram (EKG), abnormal echocardiogram (ECHO), or cardiac MRI (cMRI) findings consistent with myocarditis (as defined in Supplement 1). Cases and hospitalizations with an unknown dose number were assigned to dose 1 or dose 2 in the same proportion as the known doses: 15% occurred following dose 1 and 85% occurred following dose 2.

I was also unclear about this:

To compute crude rates per million for doses 1 and 2, our denominators included all children with at least 1 dose of any vaccination and all fully vaccinated children, respectively, as of 6/11/2021[6] to accommodate both reporting lag and a pre-defined 7-day risk window, consistent with the CDC’s analysis.

How can you use all children with at least one dose of any vaccination, given that the only vaccine that a child 12-17 years old can get is the Pfizer vaccine, given that the Moderna vaccine has not been granted an emergency use authorization (EUA) for persons under 18? Also, the Pfizer vaccine wasn’t granted an EUA for the 12-15 year age group until May 10, meaning that any data before that could only apply to 16-17 year olds, who were included in the original Pfizer EUA in December. Thus, for the 12-15 year group, this study only analyzed about five weeks’ worth of data. Surely the analysis could have been carried out to a more recent date, given that VAERS is updated weekly. Moreover, these data all apply largely to before the delta variant took off in the US to become the dominant variant driving infections. Remember, Delta is much more infectious than the original SARS-CoV-2, the coronavirus that causes COVID-19, and that changes the equation markedly. In other words, even if these data turn out to be valid (doubtful), they’re already hopelessly out of date.

This method of case ascertainment immediately led to major criticism from pediatricians, and one pediatric cardiologist in particular by the name of Dr. Frank Han:

Dr. Han’s account might be anecdotal, but evidence backs it up. Also, in fairness, Dr. Han is describing a specialist’s method of diagnosing myocarditis. In a search strategy, one doesn’t have that luxury. Still, insisting on elevated troponin does strike me as practically guaranteed to overcall cases of myocarditis. For more information, I refer you to this Twitter thread by another pediatric cardiologist, Dr. Jennifer Huang, who acknowledges that the possibility of myocarditis after vaccination is a scary one for parents. (No one, least of all I, claim otherwise. Certainly Dr. Jonathan Howard, who’s written about this extensively on SBM, does not.) She then notes:

Those questions aside, Høeg et al. reported this about cardiac adverse events (CAEs) after COVID-19 vaccination from their analysis:

A total of 257 CAEs were identified. Rates per million following dose 2 among males were 162.2 (ages 12-15) and 94.0 (ages 16-17); among females, rates were 13.0 and 13.4 per million, respectively. For boys 12-15 without medical comorbidities receiving their second mRNA vaccination dose, the rate of CAE is 3.7 to 6.1 times higher than their 120-day COVID-19 hospitalization risk as of August 21, 2021 (7-day hospitalizations 1.5/100k population) and 2.6-4.3-fold higher at times of high weekly hospitalization risk (7-day hospitalizations 2.1/100k), such as during January 2021. For boys 16-17 without medical comorbidities, the rate of CAE is currently 2.1 to 3.5 times higher than their 120-day COVID-19 hospitalization risk, and 1.5 to 2.5 times higher at times of high weekly COVID-19 hospitalization.

In the discussion, Høeg et al. add:

Our post-second-dose-vaccination rates of CAE among adolescent boys aged 12-15 was 162.2/million which exceeded the rates reported by the CDC[2,6] by 143-280% (2.4-3.8 times). Among boys age 16-17, our estimate was 94.0/million, 31.5-41% higher than the CDC estimate. For girls 12-15 years old, our rate was 13.0/million, which was 43-100% higher that the CDC’s estimate.[2,6] Among girls 16-17, our estimate was 13.4/million, which was 47-65% higher than the CDC’s estimate.

I wonder why that might be…

In search of an answer, it didn’t take long for people to notice that there are some rather…questionable…cases included in the analysis by Høeg et al., as Dr. Freedman notes about COVID vaccine associated myocarditis (C-VAM):

The authors claim to use the same methodology as the ACIP review but a brief review raises some suspicions. VAERS ID 1345283 describes a teen with chest pain and right axis deviation on EKG. The report states “no clear diagnosis but a suggestion that it sounded clinically like a viral pericarditis”. Right axis deviation is not one of the criteria used by the ACIP to determine cases of myocarditis or pericarditis (see Table 1). This is the problem with just plugging in search terms (“troponin”, “myocarditis”, etc) to VAERS and not thoroughly reviewing cases. As Ryan Marino said, “this is like thinking that a search for ‘gunshots’ on NextDoor is a way to track gun violence”.

The most glaring examples of cases that were not reviewed in detail by Hoeg et al are the cases with a comorbid infection. This represents a significant confounding variable which makes it impossible to discern with such limited data if the myocarditis was due to the vaccine or the intercurrent illness. VAERS ID 1334617 describes a positive SARS-CoV-2 PCR and VAERS ID 1361923 describes a rhinovirus/enterovirus positive PCR. The authors also include a report of a patient with EBV-positive PCR, serologies pending. These cases were likely excluded by ACIP due to these confounders.

There are other notable cases like VAERS ID 1382338 where the patient is described as encephalopathic to the point of needing intubation for airway protection. Is this a case of C-VAM or a viral infection causing both encephalitis and myocardial injury? VAERS ID 1386269 describes a patient with difficulty walking due to neurological weakness. No mention of any cardiac diagnosis. These cases were likely excluded by ACIP due to incomplete information.

On Twitter last night, Max Kennerly looked at even more:

In the comments after the preprint, Dr. David Goldberg notes:

Although the scientific question that is being address is an important one, I have concerns about the methodology used to adjudicate the outcome. In similar circumstances (e.g., the FDAs Mini-Sentinel Initiative), complex clinical outcomes like this (e.g., acute liver failure) were adjudicated independently by two experts, with a third person serving to break any ties. That seems not to have been done in this study, as there was only one cardiologist involved. Secondly, the clinical data to adjudicate the outcome of myocarditis seems to be insufficient in many cases. Although one could argue “this is the best data we have” sometimes that is not good enough. When the question is so important and politically charged, incomplete/invalid data is sometimes worse than no data. Unless the authors can have two-party adjudication with record review, and classification using standard techniques (e.g., definite vaccine-induced myocarditis, highly likely, probable, possible, not) then there are major methodological concerns with the outcome, and the overall validity of the study.

In fairness, this sort of analysis is difficult—maybe even impossible—to accomplish using VAERS data. Indeed, it is impossible to accomplish using unadjudicated data from VAERS, as Høeg et al. did. However, I like Dr. Goldberg’s point about how incomplete and invalid data can be worse than no data. Moreover, it is not difficult to see how analyzing adjudicated data, as imperfect as such data also are, would come much closer to this ideal than the analysis by Høeg et al.

The most pertinent question of all, however, comes from Roger Sehault:

Why did you choose to compare vaccine related CAE with COVID hospitalizations? Why not compare Vaccine related CAE with COVID CAE? Are we comparing apples to apples?

Why indeed? I’d also ask another question: Why did the authors not choose to compare vaccine-related hospitalizations (which could be gleaned from VAERS and the vast majority of which in children are for CAEs due to suspected myocarditis) with COVID-related hospitalizations?

Again, as many have pointed out, the clinical course of vaccine-associated myocarditis is benign, although frequently it does involve an overnight stay (or up to a few days’ stay) in the hospital for cardiac monitoring. Besides comparing apples to oranges, this sort of analysis makes vaccine-associated myocarditis sound more dangerous than the complications of COVID-19 in children. It leaves out important context that, even if its estimate of risk of myocarditis is valid (and, again, remember that VAERS by its very nature cannot provide a good estimate of any given vaccine AE), children are far more likely to die or suffer serious (and possibly life-long) complications from COVID-19. The authors acknowledge this (sort of) in the text of their study, but it’s buried in the discussion, almost as an afterthought. It’s context that is not in the abstract and therefore goes missing from all the news reports and social media posts touting this study as “scientific evidence” that children should not be vaccinated against COVID-19 because the vaccine is supposedly more dangerous than the disease. I will refer you to Jonathan Howard’s recent post in which he lists much of the recent evidence about COVID-19 vaccination in children. Spoiler alert: Although no one should take this potential AE lightly, thus far the short-term data has been consistently favorable for vaccinating individuals in the 12-17 year old age group.

VAERS: Graveyard of epidemiology (with rare exceptions)

The study by Høeg et al. is a striking example of what can happen when doctors wander into an area where they clearly lack basic critical knowledge, even doctors who are not antivaccine. After over 15 years of experience, I now expect antivaxxers to be tripped up dumpster diving in VAERS, but I do not expect it from investigators who are not antivaccine. Mea culpa. I was wrong this time. However this happened, hubris, bias, ideology, or whatever reason, Høeg et al. inadvertently did exactly the same thing antivaxxers have been doing for years: Dumpster diving in VAERS. They used better statistics than the usual antivaxxer, but the end result was the same. They should be embarrassed.

One very critical rule about research utilizing a database is that the investigators should have a strong understanding of its structure, purpose, and weaknesses. It is clear from this paper that Høeg et al. lacked an understanding of the purpose of VAERS, which is not as a database that can accurately estimate the incidence or prevalence of a given AE after vaccination. By its very design it can’t be used to do that, hence all the post-reporting adjudication and curating that go on behind the scenes at the CDC when analyzing VAERS data. The purposes of VAERS are safety signal detection and hypothesis generation. When reports of a certain AE after vaccination reach a certain threshold, there’s a safety signal, which leads the CDC to investigate first the hypothesis that the AE is actually associated with vaccination (this is where the vaccine-autism hypothesis consistently fails) and then, if there is a correlation, whether that correlation could be an indication of causation. Other databases, active surveillance system databases like the VSD and CISA, are required to test these hypotheses, because again – VAERS data are by the design of the database unsuitable for this purpose.

Surprisingly, given that I often disagree with him, one particular cardiologist gets it (mostly) right. After saying what I just said, that VAERS is only good for signal detection, not epidemiology, he notes:

This is basically where the data appear to be going. There’s a signal. It’s not yet clear how large the risk of myocarditis is in this age group. He also notes:

In fact, the dead giveaway, the aspect of this study that is the “tell” behind the authors’ purpose is that “apples and oranges” comparison between COVID-19 CAEs and hospitalization rates for COVID-19 itself. I didn’t really go into the analysis in this study of risk based on comorbidities given how the core structure of the study was so badly designed, but he’s also mostly correct about that. I would add, though, that comorbidities are important and do need investigation.

I’m sure I will anger some people with my conclusion. I don’t care. It needs to be said. The bottom line is this. This study is a dumpster dive whose only difference from what antivaxxers have been doing with VAERS since before I first started paying attention 17 years ago is that the statistics are better, but that isn’t saying much. However “provaccine” you might be (and I’ll take the investigators’ word that they are, in fact provaccine), when you find yourself doing the sort of poor quality analysis of VAERS data that antivaxxers have been doing at least since 2006, you really need to rethink your approach. Similarly, if you think that a low quality study like this that uses a database for a purpose for which it was never designed is a “bombshell” from a “dream team”, all while demanding ever more rigorous randomized trials for interventions like masking (I’m talking to you, Dr. Prasad), you might want to think about the biases that led you to such inconsistency in your standards of evidence.

Shares

Author

Posted by David Gorski

Dr. Gorski's full information can be found here, along with information for patients. David H. Gorski, MD, PhD, FACS is a surgical oncologist at the Barbara Ann Karmanos Cancer Institute specializing in breast cancer surgery, where he also serves as the American College of Surgeons Committee on Cancer Liaison Physician as well as an Associate Professor of Surgery and member of the faculty of the Graduate Program in Cancer Biology at Wayne State University. If you are a potential patient and found this page through a Google search, please check out Dr. Gorski's biographical information, disclaimers regarding his writings, and notice to patients here.

All Posts Website