I hadn’t planned on writing about the antivaccine movement again this week, so soon after having had to subject myself to yet another round of Jenny McCarthy on Larry King Live and a truly execrable Generation Rescue “study.” I really hadn’t. For one thing, there’s just so much nonsense laid down by antivaccinationists these days that it’s utterly impossible for one blogger to keep up with it all. I could write about them every single day and still not counter the sheer mass of pseudoscience, misinformation, and general ignorance that antivaccine activists spout each and every day, and because this is Autism Awareness Month lately the misinformation has been coming particularly fast and furiously. Sometimes, however, there arrives a bit of misinformation that is so egregious that it requires some response, regardless of how burned out on the topic I might be; so I guess I’ll just have to suck it up and plunge into the morass again.
The reason is that, in retrospect, I now realize that the Jenny and Jim antivaccine propaganda tour was clearly merely phase I of Generation Rescue’s April public relations offensive. In rapid succession last week, courtesy of J.B. Handley, the founder of Generation Rescue, who in order to have a couple of famous faces fronting his organization has allowed himself to be displaced, so that Generation Rescue has now been “reborn” as Jenny McCarthy and Jim Carrey’s Autism Organization (the better to capitalize on her D-list celebrity yoked to Jim Carrey’s formerly A-list (but rapidly plunging) celebrity), announced Generation Rescue’s latest initiative in a post on its antivaccine blog Age of Autism entitled Fourteen Studies? Only if you never read them.:
For me, it all started when Amanda Peet said the following in her “apology” to calling parents parasites:
“Fourteen studies have been conducted (both here in the US and abroad), and these tests are reproducible; no matter where they are administered, or who is funding them, the conclusion is the same: there is no association between autism and vaccines.”
And, don’t think Amanda Peet is alone, the mantra comes fast and furious from all sides…
These comments were driving me nuts. I’d read a majority of the studies they were referring to, I knew how bad they were, and I also knew that most journalists couldn’t even find the studies being referred to, because most weren’t even on the web!!
Several hundred hours of work later, Generation Rescue is pleased to introduce a website with a very simple purpose: to tell the truth. You will find every study in its entirety and a whole lot more right here:
Before I dive in, let me just point out right here and right now that J.B. Handley wouldn’t be able to recognize good science if it bit him on the posterior. Ditto bad science, because he simply does not have an understanding of the scientific method or the methodology involved that would allow him to distinguish good from bad science. His well-known arrogance of ignorance leads him to be very confident that he knows as much or more than the pointy-headed scientists, whom he clearly despises and for whom he clearly has little but contempt, and can thus judge the quality of complicated epidemiology and basic science, but he clearly does not. That’s why Mr. Handley’s claim that he recognized these studies to be bad science by reading them made me chuckle heartily. After all, Mr. Handley’s proven time and time again that he doesn’t understand science, the scientific method, or epidemiology, and I find the picture that pops into my mind of Mr. Handley poring over epidemiological and scientific studies, carefully making note of the methodology and picking out the flaws almost as outlandish as Jenny McCarthy’s transformation from an “Indigo Mom” to a “warrior mom” for autistic children. That’s because J.B. also understands nothing about clinical trials and especially the ethics of clinical trials, one particularly egregious example of which I should someday post about. Be that as it may, right off the bat, I knew that this “Fourteen Studies” initiative was likely to be more of the same. The claim that it took “several hundred hours of work” is probably true, but, given the results, but it clearly wasn’t time and money well spent, at least on an intellectual level. Whether it was time and money well spent in terms of producing effective propaganda remains to be seen.
Taking a general view of the Fourteen Studies, I note that it’s divided broadly into what I can’t help but refer to “studies antivaccinationists hate” (because they are for the most part large, well-designed epidemiological studies that found no association between thimerosal and autism or vaccines and autism) and “studies antivaccinationists like.” Of this latter category, the studies listed that I recognize include some small preliminary studies (which are more prone to false positives), more than a fair amount of cluelessness (the Generation Rescue “vaxed versus antivaxed” phone survey, which I’ll discuss more later), the incompetent (pretty much anything by Andrew Wakefield or Mark and David Geier), or execrable (anything published in the vanity Medical Hypotheses, which discourages the publication of data and will publish virtually article as long as it presents a provocative and speculative hypothesis, making it a “what if?” speculative journal and not a serious research journal , or the crank journal, the Journal of American Physicians and Surgeons (JPANDS), whose penchant for antivaccine pseudoscience I’ve discussed in depth before on this very blog). Heavy hitters in the world of vaccine or autism science, these are not. (Hint: If you want to be taken seriously in the world of medical science, don’t cite JPANDS. Really. Don’t. Doing so is a deal breaker. Also, citing articles from Medical Hypotheses as though they were evidence of anything other than the author’s speculative flights of scientific fancy shows more cluelessness than anything else.)
I also recognize that a lot of these criticisms are the same old dubious, exaggerated, and/or fallacious criticisms of these studies that have been floating around in antivaccine circles for a long time. I recognize them because I have read nearly all of these articles myself, both the ones Generation Rescue likes and the ones it doesn’t like, over the years, and I have seen these “criticism” True, I’m more familiar with some of these studies than others, but in fact I have even blogged about some of them, here and elsewhere. (I will link to such posts where appropriate.) In essence, what I plan to do is to set the stage by discussing the overall intellectual dishonesty that underpins Fourteen Studies and then take a few (I hope) well-chosen examples to illustrate this intellectual dishonesty. Over the coming days, it is my hope that some of my fellow SBM bloggers will take on other examples, explaining the fallacies behind the “critiques” and why the “studies antivaccinationists like” are, by and large, not very good and/or do not show what antivaccinationists think they show.
We do this because we realize that the Fourteen Studies website is going to metastasize all over the web. That is clearly Mr. Handley’s intent; he saw what he feared to be an effective public relations gambit from scientists and physicians, and he couldn’t let that go unanswered. To him, it’s not about the science, but winning the P.R. war. That’s why we know that antivaccinists will be referring journalists to it, if they have not already begun to do so. We also predict that antivaccine parents will be planting printouts of it in front of their pediatricians in order to try to persuade them that the evidence base supporting the consensus that vaccines do not cause autism is hopelessly corrupt and weak. We realize that this is exactly the purpose of Generation Rescue’s new website, which is why we believe that there should be as many rebuttals on the web to which people can be referred as possible. In fact, it is my hope that our readers will help us out by deconstructing some of the studies in the comments, and that readers who happen to be bloggers will join in this effort by applying some evidence-based criticism to the site. Unfortunately, here at SBM can’t afford to spend the hundreds of hours that Generation Rescue spent on this website in order to counter the misinformation there. We do not have the time, nor do we have the money to pay someone to do it, as Generation Rescue apparently does.
“RANKING” CRITERIA THAT GUARANTEE THE OUTCOME
First, let’s take a look at how Generation Rescue “ranks” these 14 studies. This alone should show you more than anything else why this site is propaganda, not science. Whoever put it together first ranked the studies on a scale of 40 points, made up of four 10-point subscales. The first of these subscales is described thusly:
Asked the Right Question (0-10 points):
Did the study actually contemplate the real world example of a parent vaccinating their child with 5 or more vaccines and then seeing a regression into autism afterward? Did it contemplate something close to that which could be helpful and generalized? A perfect question received a 10, a study that didn’t even contemplate the question at hand received a zero.
See what I mean? This is, of course, not the “right question” at all, other than in the minds of antivaccinationists who have already made up their minds about the answer. This is, in essence, an appeal to anecdotal evidence (“contemplate the real world example”) framed in the most “dramatic” way possible, full of inflammatory language about “five or more vaccines.” The right question is, rather, whether or not there is an association or correlation between autism and vaccination. Also, note how vague and subjective this scale is. What determines how close to the “right” question a hypothesis will be ranked, apparently, is whether it sounds good to the Generation Rescue antivaccinationist who ranked it. Certainly there do not appear to be anything resembling objective criteria, which are essential for any sort of scale like this. I also note with some amusement that one of the studies included as “Our Studies” on this website, namely a 2006 study by Raymond Palmer looking at whether proximity to coal-fueled power plants is associated with increased prevalence of autism, would get a zero on this scale, as it didn’t even look at vaccines at all! It was also a really crappy study, as I discussed extensively a few months ago during a deconstruction of Palmer’s followup study. At least Generation Rescue could have cited Palmer’s 2008 followup study instead, which, while not very good at all, was at least not as spectacularly awful as his 2006 study, which didn’t even bother to try to control for some very obvious confounders. Apparently asking the “right” question doesn’t really matter to Generation Rescue if J.B. Handley likes the answer.
Next, Generation Rescue lists:
Conflict of Interest (0-10 points):
We considered a scientist employed by a vaccine maker or a study sponsored by a vaccine maker to have the highest degree of conflict, with a public health organization (like the CDC) to be the second-worst. A conflict free study would receive a 10, a study rife with conflicts as low as zero.
This one really annoyed me, because I’ve discussed on at least a couple of occasions how rife with conflicts of interest are many of the “studies” used by antivaccine advocates to support their point of view. For example, the grossly incompetent “study” by Generation Rescue that I discussed last week and that is featured on Fourteen Studies was commissioned and performed by an organization whose raison d’être is to convince lawmakers and the public that vaccines cause autism and that “biomedical treatments” targeted at “vaccine injury” can cure autism. If that’s not a conflict of interest, I don’t know what is. The same is true of an awful study of vaccination in monkeys by Laura Hewitson (which I blogged about several months ago and that Generation Rescue widely promoted, but apparently decided to leave off of its latest propaganda initiative even though it would get a 10 out of 10 on “asking the right question” by Generation Rescue’s definition. (Maybe the criticisms stung.) After all, Laura Hewitson is not only married to antivaccine hero Andrew Wakefield’s IT director but she and her husband are complainants in the Autism Omnibus proceedings, which led me and others to wonder at the time if their “monkey study” was in fact done explicitly to provide more ammunition for the test cases under consideration by the Vaccine Court at the time.
Finally, let’s not forget Mark and David Geier, both of whom have much invested in the concept that mercury in vaccines causes autism and advocate, in essence, chemical castration with Lupron as a “treatment” for autism to help “free” the mercury from testosterone and allow it to be chelated more effectively. In fact, they took conflicts of interest to whole new levels undreamt of by even the most cynical, greedy, and amoral big pharma executive when they specially created an IRB and stacked it with their cronies in order to approve “studies” of their Lupron therapy. Any of their studies would get a zero on this the conflict of interest scale. Heck, they’d get negative scores! And all of this doesn’t even count Andrew Wakefield, who was shown by Brian Deer not only to have been an incompetent scientists, but in the pocket of trial lawyers suing vaccine manufacturers for “vaccine injury.” Worse, these are undisclosed conflicts of interest, whereas any conflict of interest listed in the “fourteen studies” was disclosed right there on the manuscript. In marked contrast, it took Brian Deer years to dig up Andrew Wakefield’s blatant conflicts of interest; Kathleen Seidel’s digging to unearth the Geiers’ abuse of the IRB process; and the digging of several bloggers to figure out Laura Hewitson’s undisclosed conflict of interest in being a plaintiff in the Autism Omnibus. If there’s one rule of thumb in medical research, it’s that undisclosed conflicts of interest correctly raise far more suspicion of bias than disclosed ones do.
Cleverly, Generation Rescue disingenuously inoculates itself from charges of picking studies with huge conflicts of interest by narrowly defining a conflict of interest the way that it wants to: As an investigator’s only being affiliated with a vaccine maker or the CDC. (Because, you know, antivaccine warriors can’t possibly ever have a conflict of interest.) Then, to guarantee that every study that it doesn’t like has an irretrievable “conflict of interest,” even if there is no pharmaceutical company connection, Generation Rescue appears to consider the mere fact of the lead investigator working for the NIH to be a severe conflict of interest. That is what is listed as the sole conflict of interest for Dr. Karen Nelson, who wrote a review article on thimerosal and autism. If that’s not enough to guarantee a “conflict of interest” for all of these “fourteen studies,” Generation Rescue also appears to define the mere fact of being funded through grants from the CDC, NIH, American Academy of Pediatrics, or the Canadian Institutes for Health Research. Scientists who have had NIH grants (such as myself) or grants from any of these other organizations know just how ridiculous considering that funding source to be a horrific conflict of interest is. After all, such grants are in general competitive grants awarded after rigorous peer review, and the NIH, for example, exercises almost no control over how researchers funded through it do their work, other than enforcing federal regulations on animal and human subjects welfare by refusing to disburse funds until the appropriate approvals are in place.
Next up, Generation Rescue ranks the ability to generalize and post-publication criticism:
Ability to Generalize (0-10 points):
This was a measure of the robustness, replicability, and usefulness of the study. Where possible, we looked to experts to help us gauge this ranking. The more robust, replicable, and broadly applicable, the higher the score.Post-Publication Criticism (0-10 points):
Was the study widely accepted in the scientific community, or was it the subject of extreme criticisms from many sources? The closer to widely accepted, the closer to 10 points.
One wonders why Generation Rescue didn’t always look to experts to gauge this ranking. After all, neither J.B. Handley, Jenny McCarthy, nor Jim Carrey are scientists. In fact, they have shown time and time again that they are clueless about science. Any conclusion by them or the other antivaccine warriors at Generation Rescue on the generalizability of a study would be highly suspect even giving them every benefit of the doubt. As part of this propaganda effort, forget it. Also, one can’t help but note that the “experts” Generation Rescue does find happen to be antivaccinationists like Boyd Haley, the former chairman of the Department of Chemistry at the University of Kentucky whose academic career imploded as a result of his having delved too deeply for too long into mercury and vaccines pseudoscience (as well as “toxic teeth” amalgam pseudoscience), or Mark Geier, whose protocol that uses Lupron to treat autism is, in my considered opinion, dangerous quackery. In other cases, the “experts” are nothing more than some of J.B. Handley’s fellow antivaccine activists with no special scientific or medical training: Sallie Bernard of SafeMinds or Deirdre Imus–or even J.B. Handley himself. Such are Generation Rescue’s “experts”! Clearly, the talent pool of “experts” at Generation Rescue is not particularly deep. One wonders why Generation Rescue didn’t draft Jim Carrey or Jenny McCarthy to do a critique of one of these articles.
Also, it’s extremely disingenuous of Generation Rescue to rank these studies on “post-publication criticism” because it is the antivaccine movement, including Generation Rescue, that accounted for the vast majority of criticisms of these articles (most were not particularly controversial among scientists). Indeed, the antivaccine movement encouraged criticism at every turn. In most cases, Generation Rescue is conflating criticisms based on ideology with scientific criticisms, which is how they keep the vaccine/autism manufactroversy alive. One can’t help but note again that, if post publication criticism were a major criterion by which Generation Rescue’s favored studies should be judged, Andrew Wakefield’s 1998 Lancet study was disavowed by 10 of its original 13 authors and ultimately utterly repudiated by the scientific community–and deservedly so! Yet there it is on the “Our Studies” page of Fourteen Studies. Apparently only criticism by the antivaccine movement counts as a negative to Generation Rescue. Criticism by scientists? Apparently not so much.
ON TO SOME OF THE “FOURTEEN STUDIES”
Mady Hornig’s 2008 study
Even though it was obvious from the start that this website is every bit as much propaganda as the Green Our Vaccines rally in Washington, DC last year, still I wanted to see what Generation Rescue said about some of the studies. So I picked a few that I am quite familiar with. First, off, I noticed that Mady Hornig’s study from last fall that tried to replicate Andrew Wakefield’s original Lancet study that implicated the MMR vaccine in the pathogenesis of “regressive” autism and enterocolitis. You might remember this one too. You might also recognize that all Generation Rescue did was to regurgitate old attacks by Andrew Wakefield, SafeMinds, and the National Autism Association on it. You may also recall that I blogged about this study shortly after it was published and showed you why these criticisms were, scientifically speaking, a load of horse hockey. Suffice it to say, they have not aged well.
The 2009 Italian study
Another study included on the list is an Italian study that came out this year by Tozzi et al, which was published in the February issue of Pediatrics and entitled Neuropsychological Performance 10 Years After Immunization in Infancy With Thimerosal-Containing Vaccines. The study was done in Italy, and one of its great advantages is that the amount of thimerosal to which the infants were exposed was actually known, unlike many epidemiological studies, where sometimes the dose of thimerosal (and therefore mercury) has to be estimated or inferred from the vaccine schedule at the time.
The reason is that the children studied were children who had taken part in a randomized study of two different diphtheria-tetanus-acellular-pertuss (DTaP) vaccines, one that contained thimerosal and one that did not:
In 1992-1993, 15,601 healthy, 2-month-old infants were enrolled in the Italian Trial on Pertussis Vaccines. 18-20 In this trial, infants were selected from the general population in 4 of Italy’s 20 regions (Fig 1) and were assigned randomly to receive, under double-blind conditions, 3 doses of 1 of 4 vaccines, 2 of which were DTaP vaccines from 2 different manufacturers. One DTaP vaccine contained 50 μ g of thimerosal (or 25 μ g of ethylmercury) per dose, and the other was thimerosal-free (2-phenoxyethanol was used as preservative). The 3 doses of DTaP vaccine were administered at 2, 4, and 6 months of age. To comply with Italy’s vaccination schedule, all children also received 3 doses of hepatitis B virus vaccine (child formulation), each of which contained 25 μ g of thimerosal (or 12.5 μg of ethylmercury), at 2, 4, and 12 months of age, and a fourth dose of diphtheriatetanus vaccine, which contained 50 μg of thimerosal (or 25 g of ethylmercury), at 11 months of age…Therefore, in the first 12 months of life, the cumulative intake of ethylmercury, the mercury metabolite of thimerosal, was 137.5 μg for the children who were assigned randomly to receive the DTaP vaccine that contained thimerosal (“higher intake group”) and 62.5 μ g for those who received the thimerosal- free DTaP vaccine (“lower intake group”).
Ten years later, the children from this study living in the Veneto Region were studied. 1,403 children (697 belonging to the high thimerosal group and 706 belonging to the low thimerosal group) were recruited and subjected to a battery of eleven neurodevelopmental tests that produced a total of 24 neuropsychological outcomes to assess their development. The results were unsurprising and very much like the results of a study of thimerosal-containing vaccines as a risk factor for neurodevelopmental disorders other than autism that was published a year and a half ago. Most outcome measures showed no difference between the low and high thimerosal group, and the ones that did were small and entirely compatible with random chance due to multiple comparisons.
I knew at the time exactly what antivaccine activists would probably say when and if they get around to attacking this study, and, in fact, that’s basically the criticism on Fourteen Studies. (Most depressingly, Generation Rescue even reprints J.B. Handley’s scientifically illiterate criticism of the study.) In any case, the main criticism is that there was not a control group receiving no thimerosal. True enough. The authors themselves make that very point. However, if thimerosal in vaccines were associated with autism, one would not expect that it would be different than any other toxin associated with an abnormality, disease, or condition in that it would be expected that the chance of autism or neurodevelopmental disorders would increase with increasing dose.
The second argument that I thought advocates would try to make but that Generation Rescue actually did not make is that the dose-response curve for mercury and autism has a plateau, and that plateau is below 62.5 μg, hence the lack of difference between the two groups. There’s just one problem with that argument. An exposure to 62.5 μg, to which the low exposure group was exposed, corresponds to roughly the total dose of mercury in thimerosal to which American infants were exposed in 1989—before the alleged “autism epidemic.” Even if mercury does indeed cause autism and there is indeed a plateau in the dose-response below a dose of 62.5 μg, that would not be consistent with the antivaxers’ other pet claim, that an autism epidemic started in the 1990s because of the increasing amount of thimerosal exposure due to vaccines. That couldn’t have happened if a dose of thimerosal less than 62.5 μg maxed out the risk of autism, because 62.5 μg below the baseline exposure before the alleged “autism epidemic” started. The two claims (that of an autism epidemic in the 1990s due to increasing amounts of thimerosal in vaccines versus that of an effect that maxes out before a 62.5 μg cumulative dose of mercury from thimerosal) are mutually contradictory. I suppose antivaccinationists could postulate a threshold effect that doesn’t occur until a dose above 137.5 μg. Unfortunately for them, then they would have the problem of how long they ranted that any mercury was toxic and any mercury was unacceptable, not to mention the–shall we say?–inconvenient epidemiological evidence that autism rates did not plummet back to 1980s levels after 2001, which was when thimerosal was removed from most childhood vaccines and mercury exposure from vaccines plummeted to well under 62.5 μg.
Finally, the most ironic thing about this study is the question of why autism prevalence was so low among the Italian population studied. Of course, as Generation Rescue points out, all of the subjects received at least 62.5 μg mercury from their vaccines; so by antivaccinationist logic there should have been a much larger prevalence of autism, particularly among the Italian children who received 137.5 μg mercury. There wasn’t. The prevalence was much lower than in the U.S., which also rather undermines the claims Generation Rescue made in its recent “vaccines by nation” nonsense. So much for “mercury poisoning,” at least in this case.
The 2008 Schechter study
Antivaccinationists really, really hate this study because it looked at a very specific prediction that naturally follows from the hypothesis that mercury in the thimerosal preservative in vaccines was a major cause of the “autism epidemic.” I say “was” because by early 2002, thimerosal had been removed from the vast majority of vaccines, and this allows the testing of the hypothesis. Specifically, the prediction is that, if thimerosal in vaccines was indeed a major cause of autism, autism incidence should have plummeted within a few years after it was removed. This study showed that there is no evidence that the apparent rise in autism is even plateauing yet in California. In a truly fun bit of additional irony, this study even uses the California Department of Developmental Services (CDDS) database, to which David Kirby has often referred as a good way of looking at whether autism rates have started falling since thimerosal was removed from most vaccines. He has also since been moving the goalposts as to when he would expect autism rates to begin falling if the thimerosal hypthesis is true, going first from 2005 to 2007 and, most recently, to 2011.
In order to ask the question of whether autism rates had declined, Schechter and Grether examined data for clients with active status reported from January 1, 1995 to March 31, 2007. Using careful statistical analyses, they used two approaches to measure the occurrence of ASD during this period. The second approach, in which ASD prevalence was determined in the 3 to 5 year old cohort, is perhaps the most informative. It shows a continuing increase in autism prevalence without even a blip or decrease in the rate of increase after 2002. Indeed, showing the skill of some bloggers to analyze the same data, the money figure in the paper (Figure 3) looks almost exactly the same as the graph prepared in early 2007, a continually increasing curve since 1995. This result is not only consistent with multiple other published and unpublished studies, including the famous (or, if you’re an antivaccinationist, infamous) Danish and Canadian studies, but it is about as unambiguous evidence as can be obtained from a database like the CDDS database. Indeed, despite the known limitations of the use of this database, it is an excellent example of proponents of a “mercury injury” hypothesis of autism being “hoisted by their own petard,” so to speak. Even more amusingly, one of the criticisms of the study was written by Deirdre Imus, who has been known to regularly lay down some of the more–shall we say?–easily refuted antivaccinationist canards, all cloaked in “concern for the children,” is one of the “experts” refuting this study, while Boyd Haley wrote a long rant that is mostly a non sequitur.
The complaints about this study from antivaccinationists can be boiled down to one: That mercury hasn’t been completely eliminated from childhood vaccines. In other words, even though mercury has been removed from all mandated childhood vaccines other than the flu vaccine (and there are thimerosal-free versions of the flu vaccine available), leaving only trace thimerosal in the current set of vaccines, according to antivaccinationists that’s not enough. To them, mercury is just that toxic, and even the trace levels left in the vaccines are enough to keep the “autism epidemic” moving in a steadily upward direction. This is an utterly fallacious criticism for exactly the same reasons the criticisms of the Italian study listed above are similarly fallacious. Once again, remember that the “mercury hypothesis” states that an “autism epidemic” began in the late 1980s/early 1990s after the government expanded the number of mandated vaccines containing thimerosal. A consequence of that claim is that we should expect late 1980s/early 1990s levels of autism to be associated with a certain dose of mercury in vaccines, which, as you remember, was approximately 60-75 μg in 1989–by antivaccinationist Mark Blaxill’s own estimates! With now thimerosal removed from all vaccines save the flu vaccine (which many children still don’t get anyway), all that is present in the total vaccine dose received by children is a level of mercury less than what children received in 1989. So, in the world of antivaccinationists, lowering the level of total mercury received in vaccines by children to a level lower than what children received 20 years ago (before the “autism epidemic”) is still too toxic. (Consistency has never been the antivaccine movement’s strong suit.) Unfortunately, by the antivaccinationists’ own arguments, if the graph that Mark Blaxill likes to show represents not just correlation but causation, the dose-response curve for mercury must be such that the level in 1989 resulted in a relatively low level of autism; therefore, lowering it to levels below those seen in 1989 should result in a dramatic decrease in autism rates–again, using the mercury militia’s own numbers.
It didn’t. Stick a fork in the mercury-thimerosal-autism hypothesis. It’s quite done.
Thompson et al, 2007
This study did not specifically look at autism, but was the first publication of a larger study looking at neurodevelopmental outcomes after vaccination using data from the Vaccine Safety Datalink (VSD) database, a massive database overseen by the CDC and maintained through several large HMOs that collects data on vaccination outcomes. The first publication, Thompson et al, looked at whether there were adverse neurodevelopmental outcomes other than autism that could be identified as being correlated with thimerosal-containing vaccines. A second study looking specifically at autism using the VSD is in the works.
The authors found weak associations between thimerosal and developmental outcomes that were most consistent with being due to chance. It is true that there were some negative correlations found that achieved statistical significance. However, when running 42 tests, it would be shocking if there were not a few anomalous associations. What makes the study authors confident that the findings are anomalous is that they were divided roughly equally in both directions, good and bad. It is true that there were some negative correlations found that achieved statistical significance. When running 42 tests, it would be shocking if there were not a few anomalous findings. What makes the study authors fairly confident that the findings are anomalous is that they were divided roughly equally in both directions, good and bad. Consequently, if antivaccinationists are going to insist that the correlation, for example, with increasing mercury exposure and poorer performance on the GFTA-2 measure of speech articulation test is real and due to likely causation, then it must also accept the findings of a beneficial association between mercury and identification of letters and numbers on the WJ-III test, as there is no scientific reason to reject it. Even Autism Speaks, one of the “guest commentators” in Fourteen Studies, concedes this but tries to latch onto these few findings as being somehow significant.
Another important aspect of this study is that Sallie Bernard of SafeMinds, was a panelist on the consulting board that helped design this study. She pulled out when the results weren’t what she had been hoping for and has been criticizing the study ever since, up to and including being one of the “guest commenters” on this article for Generation Rescue. Her complaint is the most ridiculous of all, specifically that the study
comprised children who were least likely to exhibit neuropsychological impairments. Specifically, children with congenital problems, those from multiple births, those of low birth weight, and those not living with their biological mother were excluded. The sample was skewed toward higher socioeconomic status and maternal education — factors that are associated with lower rates of neurobehavioral problems and higher intervention rates and that were not measured.
Ms. Bernard is too scientifically illiterate to realize that that’s because the hypothesis being studied was that mercury in vaccines is associated with neurodevelopmental problems in normal children. Remember, in virtually all of the testimonials about regressions after vaccines, parents insist that their child was absolutely, positively normal before those horrible pediatricians pumped them full of those nasty, toxic, autism-causing vaccines. In fact, this study was looking at exactly the “right question.” Also (and arguably more importantly for the study design) children with congenital problems and of multiple births or of low birth weight have a higher baseline level of neurodevelopmental problems. Including such children would have (1) made it difficult to determine whether thimerosal actually was associated with neurodevelopmental difficulties because the effect size could well be dwarfed by the baseline levels of neurodevelopmental disorders in this population and (2) introduced confounders that would have been very difficult to control for in the final statistical analysis.
In any case, Ms. Bernard is being intellectually dishonest to an extreme. She participated in the design of the study but never criticized its design until after its results contradicted her expectations. Does anyone think she would have had a problem with this study if it had come back less resoundingly negative, with even a bit of wiggle room? Such is the intellectual dishonesty of the antivaccine movement and a major reason why it was always a bad idea for the government to include antivaccinationists in the the decision-making process about anything in the name of “inclusion” and the hopes that if they had a stake in the outcome they would be more constructive. The government hoped to shut up antivaccinationists by including them on such panels; it should have realized that they would turn on the government as soon as it didn’t get its way or as soon as scientific studies in whose design they participated failed to yield the results craved by the antivaccine movement.
AN EXAMPLE OF “GOOD” VACCINE STUDIES, ACCORDING TO GENERATION RESCUE
Before I conclude, I was curious to note what Generation Rescue apparently considers to be “good” studies. I already mentioned some of the awful studies, such as those by Andrew Wakefield or Mark and David Geier, that Generation Rescue likes. One study from 2009 included on the list is one I haven’t seen before, and I may well write about it on its own, as it makes a provocative but, in my opinion, ultimately highly doubtful conclusion.
One constant refrain of antivaccinationists is that there should be a study of “unvaccinated” versus “vaccinated” children. Never mind that they never quite state the hypothesis that such a study would study and sometimes even suggest utterly unethical studies, such as a randomized double-blind, placebo-controlled study. Such a study, although methodologically rigorous, would be completely unethical because it would leave the control group unprotected against vaccine-preventable diseases. Nonetheless, that doesn’t stop Generation Rescue from listing as number two its utterly risible and badly executed telephone study from 2007.
Kevin Leitch did an excellent job of deconstructing the numerical shenanigans in the poll that lead GR to boldly claim things like:
All vaccinated boys, compared to unvaccinated boys:
- Vaccinated boys were 155% more likely to have a neurological disorder (RR 2.55)
- Vaccinated boys were 224% more likely to have ADHD (RR 3.24)
- Vaccinated boys were 61% more likely to have autism (RR 1.61)
Older vaccinated boys, ages 11-17 (about half the boys surveyed), compared to older unvaccinated boys:
- Vaccinated boys were 158% more likely to have a neurological disorder (RR 2.58)
- Vaccinated boys were 317% more likely to have ADHD (RR 4.17)
- Vaccinated boys were 112% more likely to have autism (RR 2.12)
(Note: RR means “relative risk,” which in this poll is the ratio of the percentage of the condition of interest in the group of interest to the percentage found in the control population, in this case the allegedly “completely” unvaccinated.)
I’m not going to go over the same ground that Kevin Leitch so ably covered when the poll first came out, specifically the numbers. Kevin did a good job with the details showing that the “finding” are not nearly as impressive as they are represented to be. Particularly amusing is the observation that, for several of the groups, the “partially vaccinated” (whatever that means; it’s not defined) had apparently higher numbers of parents reporting autism or ASD but with parents of fully vaccinated children reporting numbers the same or lower than the unvaccinated, leading him to drolly observe:
There’s no getting away from this. This is a disaster for Generation Rescue and the whole ‘vaccines cause autism’ debacle. Generation Rescue’s data indicates that you are ‘safer’ from autism if you fully vaccinate than partially vaccinate. It also indicates that across the spectrum of autism, you are only 1% more likely to be autistic if you have had any sort of vaccination as oppose to no vaccinations at all – and thats only if you are male. If you are a girl you chances of being on the spectrum are less if you have been vaccinated! Across both boys and girls, your chances of being on the spectrum are less if you have received all vaccinations.
Of course, what the results actually suggest is a reporting bias in the phone survey in which parents whose children developed autism or an ASD and who believed vaccines might have been responsible stopped vaccinating, thus falling into the “partially vaccinated” group. Kevin’s also kindly converted the locked PDF with the raw data supplied by J. B. Handley into an Excel spreadsheet that allows more easy analysis of the figures. Some commenters ran some statistical tests on the raw data for various groups. Not surprisingly, the results were not statistically significant in nearly all cases. I looked at a few groups myself and did a few chi squared tests, and I failed to find any statistically significant differences either. I will admit that I did this only to sample the data; I have better things to do with my time than an exhaustive analysis.
Given that background, I’d like to step back a bit and look at the big picture. This phone poll is fatally flawed as a medical study for several reasons. First, note how there are now more diagnoses being looked at. Given Generation Rescue’s previous concentration on autism almost exclusively, I found it striking that ADD and ADHD were in the mix. In one row, ADD, ADHD, autism, and ASDs were all combined into one group, even though there is no etiological or logical reason to do so. Generation Rescue refers to this as “neurologic” diagnoses, but then why did the survey limit itself to just the above? After all, mental retardation is a neurological diagnosis. Seizure disorders are neurological diagnoses. It makes very little sense.
In reality, the whole enterprise was nothing more than one huge case of doing multiple comparisons and seeing if anything shakes out. Remember, at the 95% confidence level there’s still a 5% chance that any seemingly “positive” result that is found is in reality due to chance alone. The more groups looked at and compared, the more chances of a spurious result that isn’t “real.” There are statistical methods for controlling for multiple comparisons, but there’s no evidence that I can find that they were done for this poll. My best guess is that this survey was nothing more than a sloppily conducted fishing expedition where they didn’t even bother to control for multiple comparisons. Moreover, doing what is in essence subgroup analysis is dubious when only around 6% of the children (a reassuring figure, actually) were totally unvaccinated, as it makes the numbers of unvaccinated in many of the subgroups too small to be statistically useful (not that that stopped Generation Rescue from slicing and dicing the group any which way it could to extract more probably spurious “correlations”). I’m not even entirely convinced that the 991 unvaccinated children supposedly identified by this poll represent a big enough sample. Moreover, the poll concludes that there is no increased risk of autism or other ASDs in girls due to vaccination, which makes me wonder if J. B. will change his tune and urge girls, at least, to be vaccinated.
Another problem with the poll is how the vaccinated group was divided into “partially vaccinated” versus “fully vaccinated.” For one thing, the definition of “fully vaccinated” would not be the same in all age groups, given that the recommended vaccination schedule changes periodically based on new recommendations from the CDC and the American Academy of Pediatrics. Second, “partially vaccinated” would encompass a huge range of possibilities, from children who only received one or two recommended vaccinations to those who received all but one. It’s an almost meaningless distinction, particularly in a phone survey. The only sort of study for which separating the vaccinated into two groups like that might be useful is one in which investigators can review the vaccination records of the subjects polled and know who got exactly what vaccines. No doubt J. B. was hoping to find some sort of dose-response curve, with increasing levels of neurologic diagnoses as one goes from unvaccinated, to partially vaccinated, to fully vaccinated. That makes it all the more hilarious that the results show in many groups equal percentages of diagnoses in the unvaccinated and fully vaccinated groups, with the peak percentages reported being in the partially vaccinated group and suggests that the real comparison that should have been made was one that wasn’t: between the completely unvaccinated and patients who had gotten any vaccines (partially vaccinated + fully vaccinated). Of course, again, this also suggests a reporting bias, where parents who think their children’s problems stem from vaccines stop vaccinating. Or it could just be spurious. What it does suggest is that there is a serious problem with the poll.
The biggest problem, however, that makes me doubt this survey is the questionnaire and how the study was conducted. J. B. makes a big deal about how their methodology “mirrored” the methodology that the CDC used to establish estimates of autism prevalence. The two are only superficially comparable. The CDC used very simple methodology:
The surveys were independently conducted but both were conducted during the same time period, 2003 to 2004. Both were based on a nationally representative sample of non-institutionalized U.S. children and in both surveys parents or guardians of the sampled children were asked about a range of different health issues.
Autism prevalence was estimated from the question asking parents if they were ever told by a doctor, or other health care providers, that their child had autism.
Leaving aside that Generation Rescue doesn’t describe how their sample was chosen, what measures were taken to make sure it was representative, and what the response rate was, compare this to the questionnaire that Generation Rescue tried to get parents to answer. Here’s a sample:
5) If this child has ever been diagnosed with asthma, juvenile diabetes, autism, Asperger’s Syndrome, ADD, ADHD, or PDD-NOS, press 1 (continue to next Q)
Otherwise, press 2 (skpt to closing language “B”)6.) OK, then, let’s go through each condition one at a time.
Has this child been diagnosed…
With Asthma?
Yes, press 1
No, 2
Not sure? 37.) With Juvenile Diabetes?
Yes, press 1
No, 2
Not sure? 3.8.) With Autism?
Yes, press 1
No, 2
Not sure? 3.9.) With Asperger’s?
Yes, press 1
No, 2
Not sure? 3.10.) With ADD?
Yes, press 1
No, 2
Not sure? 3.11.) With ADHD?
Yes, press 1
No, 2
Not sure? 3.12.) With PDD-NOS?
Yes, press 1
No, 2
Not sure? 3.
PDD-NOS? How many parents know what PDD-NOS is? (It’s pervasive developmental disorder, not otherwise specified, by the way.) I didn’t know what a PDD-NOS was until I became interested in the vaccine-autism hysteria. The only parents who are going to know what that is are the ones whose child has a diagnosis. Heck, even though most parents know what autism is, a lot of parents don’t know what Asperger’s is unless their child has it.
Moreover, the CDC study produced estimates of autism prevalence that were consistent with previous studies. In contrast, J. B.’s survey produced estimates of autism and ASDs of 3% in the aggregated data. That’s 1 in 33, approximately 5 times more prevalent than the usually cited estimate of 1 in 166 or 1 in 150. This, too, suggests reporting bias, where parents who have a child with autism or an ASD will be more likely to complete this survey. Moreover, SurveyUSA is known for asking very concise questions that have been known in the past to produce divergent results using automated telephone polls, which have a number of problems, not the least of which is a much lower response rate than traditional polls. Again, the issue of reporting bias comes up. That’s not to say that it’s not possible to do accurate polls with automated technology, but asking about health problems is difficult, leading me to take any such poll done looking for correlations between vaccines and anything with a huge grain of salt. Moreover, no evidence about response rates or how many parents responded “not sure” for each question is presented, nor is any evidence to show that the sample chosen is representative. In any case, Prometheus has done an excellent job discussing how a true “vaccinated versus unvaccinated” study could be done with both a reasonable degree of scientific rigor and ethically as well. Unfortunately, it would take a minimum of 20,000 children:
A study of autistic children in the 3 – 6 year age range would need over 683,000 children in each group to detect a 10% difference in vaccination level. It would need nearly 22,000 in each group to detect a 50% difference. With a predicted number of 110,000 autistic children in that age range, that is a sizeable fraction of all autistic children.
A more manageable study – one with 10,000 children in each arm (which is still a HUGE study!) – would only be able to tell the difference between the national average of 0.3% unvaccinated in the non-autistic group and 0.1% unvaccinated in the autistic group (at the specified levels of confidence). If the difference is smaller than that, the results would be considered negative (i.e. that there is no effect of vaccination). For reference, a study with 1,000 children in each arm would show statistical significance (at our specified level of confidence) only when the autistic group was below 0.01% unvaccinated or above 1.7% unvaccinated.
Of course, we wouldn’t have to just look at unvaccinated vs fully vaccinated with this study, which is a large part of its superiority. We could look at a dose response of vaccination – to see if it really is “too many” – as well as the age at youngest vaccination – to see if it really is “too soon”. In fact, a few studies have already looked at those issues and found that there is no difference between the autistic and non-autistic groups. I suspect this is the reason the folks pushing to “put on a study” want to look at vaccinated vs unvaccinated – they hope that the numbers will be different (or, at least, not as definitive) the other way round.
This would, of course, be a massively expensive study to do as well and would funnel more research money to dubious studies of highly unlikely hypotheses.
CONCLUSION
Given that Generation Rescue paid $200,000 to do its poll, if I were J.B., I’d be asking for my money back. On the other hand, compared to how much money Generation Rescue is raking in hand over fist, thanks to Jenny McCarthy, who’s been hitting the pro wrestling circuit, doing celebrity poker events with Lance Armstrong, holding fundraisers with Hugh Hefner and Britney Spears and making numerous appearances to raise money for Generation Rescue, I’m guessing that these days $200,000 is pocket change. Moreover, only J.B. knows how much money was spent on the Fourteen Studies website, but I’m guessing it was quite a bit. The slickness of the website, coupled with all the TV appearances, and celebrity fundraisers, all of which coopt the message of “autism awareness” and conflate it with the antivaccine activist agenda, bespeak large amounts of money flowing into the antivaccine propaganda effort. Wouldn’t it be wonderful if Generation Rescue actually spent its money on real research, instead of propaganda like the Fourteen Studies website?
Don’t hold your breath waiting for that to happen.
That’s because, unfortunately, it’s not about science, evidence, or reason to Generation Rescue. It never has been; it is not now; and it never will be. No, it’s all about winning, which means that the evidence showing that vaccines are not associated with autism must be attacked, because the AAP, CDC, and Paul Offit are the “bad guys.” That’s why this Fourteen Studies website is nothing more than pure propaganda; indeed, it’s every bit as much propaganda as Jenny and Jim’s media tour to promote McCarthy’s latest paean to antivaccine pseudoscience and autism quackery or the “Green Our Vaccines” march last spring. I also note that there are far more than “14 studies” showing no association between vaccines and autism. Indeed, Mady Hornig cited twenty just looking at MMR and autism alone. That defenders of vaccines have chosen “fourteen studies” as a talking point was not a good idea; it was a strategy that made a website like Fourteen Studies possible. That, alas, was a failing of scientists as communicators.