I have a love-hate relationship with epidemiology.
On the one hand, I love how epidemiology can look for correlations in huge sample sizes, sample sizes far larger than any that we could ever have access to in clinical trials, randomized or other. I love the ability of epidemiology to generate hypotheses that can be tested in the laboratory and then later in clinical trials. Also, let’s not forget that epidemiology is sometimes the only tool available to us that can answer some questions. Such questions generally involve hypotheses that can’t be tested in a randomized clinical trial because of either ethical concerns or others. A good example of this is the question of whether vaccines cause autism. For obvious ethical reasons, it’s not permissible to perform a randomized clinical trial in which one group of children is vaccinated and one is not, and then outcomes with respect to neurodevelopmental outcomes, such as autism and autism spectrum disorders, are tracked in the two groups. The ethical concern with such a study, of course, is the potential harm that would be likely to come to the unvaccinated control group, children who would be left unprotected against common and postentially deadly communicable diaseases.
On the other hand, epidemiology is one of the messiest of sciences, and epidemiological studies are among the most difficult in all of science to perform truly rigorously. The number of factors that can confound are truly amazing, and as a result, it’s very, very easy for an epidemiological study to detect apparent correlations that are either spurious or appear much stronger than the “true” correlation. There can be confounding factors beneath confounding factors wrapped in more confounding factors, the relationships among which are not always apparent. Not infrequently, a condition can appear to be correlated with, for instance, an environmental factor, but in reality that environmental factor and the condition both correlate with a third, unknown confounder. Worse, epidemiologists know that correlation does not necessarily equal causation, but the general public, for the most part, does not, which is why, when anti-vaccine activists, for instance, point out to a rising autism prevalence and then point out that autism prevalence started rising around the same time the vaccine schedule was expanded, to the average layperson the argument sounds compelling. As a result, the design of an epidemiological study is paramount in order to account for or minimize such factors. That’s why I always said I can’t be an epidemiologist. Even though I was very good at math in college, the statistics still made my brain hurt, and I don’t have the patience for the messiness of trying to account for all the possible confounding factors.
However, for all their strengths and flaws, epidemiological studies are an integral part of science-based medicine. They are used to identify predisposing factors to diseases and conditions, environmental contributors to disease, and adverse reactions to drugs, among many other useful pieces of data. That’s why, from time to time, I like to examine epidemiological studies, particularly if they’re epidemiological studies that are getting a lot of press.
The use and abuse of autism epidemiology studies
For instance, studies like this one described in a story in the Los Angeles Times on Friday entitled Proximity to freeways increases autism risk, study finds: More research is needed, but the report suggests air pollution could be a factor:
Children born to mothers who live close to freeways have twice the risk of autism, researchers reported Thursday. The study, its authors say, adds to evidence suggesting that certain environmental exposures could play a role in causing the disorder in some children.
“This study isn’t saying exposure to air pollution or exposure to traffic causes autism,” said Heather Volk, lead author of the paper and a researcher at the Saban Research Institute of Children’s Hospital Los Angeles. “But it could be one of the factors that are contributing to its increase.”
Another news story describes the study thusly:
Naturally, the anti-vaccine crank blog Age of Autism is expressing interest in this study. The interview, of course, is truly execrable, particularly the bubble-head host saying “some people” think it’s the MMR, as though the anti-vaccine movement who demonize the MMR as a cause of autism have the slightest clue what they are talking about, and mentioning Jenny McCarthy as though she were anything other than a bad actress with a Google University education that led her to believe that vaccines caused her son’s autism.
Before I get to the study itself, one thing I always wondered about is why anti-vaccine groups are so keen on buying every study that purports to show an environmental component that might predispose to autism. Naturally, they often manage to related it back to vaccines, as they tried to do when there was a bad study purporting to show a link between proximity to power plants whose emissions contained mercury and autism. At least with that study, the potential relationship was fairly obvious in that a large contingent of the anti-vaccine movement still blames mercury in the thimerosal used in childhood vaccines vaccines as a preservative (in the U.S., at least) until the end of 2001. True, the chemical form of mercury in the environment is different than thimerosal, with different implications for toxicity and metabolism, but to the anti-vaccine mercury is mercury is mercury, and it’s equally evil.
Personally, I suspect that the reason anti-vaccine groups glom onto any study that purports to identify an environmental contributor to autism is guilt by association. If, “reason” anti-vaccine activists, this pollution or pesticides are implicated as a cause of autism, then maybe vaccines do too, and they aren’t so crazy after all for saying so. I’m sure regular readers of this blog can see the flaws in that logic. Also, many anti-vaccine activists are also advocates of a wide variety of “autism biomed” quackery, which can range from chelation therapy, to supplements, to hyperbaric oxygen chambers, and even to stem cell quackery. If pesticides, chemicals, environmental pollutants, or whatever can be implicated as a cause or contributor to autism, each new “cause” opens up a new panoply of “autism biomed” quackery to be tried.
So let’s get to the meat of the study itself.
The CHARGE study, proximity to freeways, and autism
My usual loghorreic introduction complete, let’s take a look at the study itself. It’s by a group of investigators from the University of Southern California, and its lead investigator was Heather Volk, the woman who appeared on the above FOX News segment, and the title of the article is Residential Proximity to Freeways and Autism in the CHARGE Study. Conveniently, the anti-vaccine crew at Age of Autism has posted an actual PDF of the study. One of these days, they’re going to get into trouble for doing that, but for now it allows anyone who doubts my analysis to go straight to the source. Let’s take a look at the abstract, for those of you who don’t want to read the whole paper:
Background: Little is known about environmental causes and contributing factors for autism. Basic science and epidemiological research suggest that oxidative stress and inflammation may play a role in disease development. Traffic-related air pollution, a common exposure with established effects on these pathways, contains substances found to have adverse prenatal effects.
Objectives: To examine the association between autism and residence proximity, during pregnancy and near the time of delivery, to freeways and major roadways as a surrogate for air pollution exposure.
Methods: Data were from 304 autism cases and 259 typically developing controls enrolled in the Childhood Autism Risks from Genetics and the Environment (CHARGE) Study. The mother’s address recorded on the birth certificate and trimester specific addresses derived from a residential history obtained by questionnaire were geo-coded and measures of distance to freeways and major roads were calculated using ArcGIS software. Logistic regression models compared residential proximity to freeways and major roads for autism cases and typically developing controls.
Results: Adjusting for sociodemographic factors and maternal smoking, maternal residence at the time of delivery was more likely be near a freeway (≤309 meters) for cases, as compared to controls (odds ratio (OR), 1.86, 95% confidence interval (CI) 1.04-3.45). Autism was also associated with residential proximity to a freeway during the third trimester (OR, 2.22, CI, 1.16- 4.42). After adjustment for socio-economic and demographic characteristics, these associations were unchanged. Living near other major roads at birth was not associated with autism. Conclusions: Living near a freeway was associated with autism. Examination of associations with measured air pollutants is needed.
There are a number of problems with this study, as you might imagine. First off, in epidemiological terms, it’s rather small for a study of this nature. On the other hand, it does have the strength that each child was examined individually for various factors; in other words, individual-level data were used rather than ecological (group-level) data, the latter of which has a huge tendency to result in false positives. Worse, there are a number of inconsistencies in the results, but in particular one glaring confounding factor that apparently failed to be taken into account, at least as far as I can tell.
First, let’s look at the anomalous findings. The main finding of this study is that being within 309 m of a freeway was associated with a barely statistically significant odds ratio of being diagnosed with an autism spectrum disorder of 1.86 (95% confidence interval: 1.03 to 3.45, Table 2). As noted in the abstract, autism was also correlated with residential proximity to a freeway during the third trimester (OR, 2.22, CI, 1.16-4.42). Note that the confidence interval almost encompasses 1.0. For purposes of this study, freeways were defined as U.S. interstate highways or state highways. You might also wonder why 309 m was chosen as the distance cutoff. I wondered that too when I first read news accounts of this study. My first thought was that 309 m was equivalent to 1,000 ft., and so it is–approximately. However, it’s not close enough to explain why 309 m was chosen as the cutoff, given that 309 m equals 1,013.8 ft., and 1,000 ft. equals 304.8 m. On the surface, it looked like a bit of the old data dredging to find a distance that gives a “significant” result. However, this is how the manuscript described the rationale for choosing the distances:
We examined the distribution of distance from the nearest freeway among subjects in our study and determined exposure cut-points to define the closest 10% (<309m), subsequent 15% (309m-647m), and following 25% (647m-1419m) as exposure groups. The remaining 50% (>1419m) served as the reference group in our analysis. Living within 309m of a freeway at birth was associated with autism (odds ratio (OR) 1.86, 95% confidence interval (CI) 1.04-3.45). This association was not altered by adjustment for child gender or ethnicity, maximum education in the home, maternal age, or maternal smoking during pregnancy (Table 2). When we categorized our distance measure into deciles, only the top 10% corresponding to the <309m category above, showed evidence of an increased autism risk compared to those living farthest from the freeway (lowest decile, >5,150m) (unadjusted OR=2.48, 95%CI 1.17-5.39).
Now, here’s the anomaly. This very same study failed to find any correlation with living near what is defined as a “major road,” specifically a state highway, interstate highway, or major arterial. This was true even though the distances from a major road of the closest 10%, 25%, and 50% in this study were 42 m, 96 m, and 209 m, respectively, all much closer. If pollution is the common factor that accounts for the observation of a relative risk for being diagnosed with autism of 2-fold greater than in children farther away from freeways, then these results are meaningless without knowing the levels of pollution at 42 m from a major road compared to 309 m from a freeway. The authors try to explain this anomaly away by doing some hand waving about the higher traffic volumes carried by Southern California freeways, which might well be true, but proximity matters. If proximity to a freeway is a surrogate for exposure to pollution (or a certain subset of pollutants), is it so unreasonable to expect that being more than seven times closer to a major road compared to a freeway would result in exposure to as much pollution? I don’t know the answer to this, although this sort of data is known. For instance, it’s known that resuspension by Los Angeles Interstate 405 significantly increases atmospheric particle numbers and metal concentrations but that both return to urban background levels through dry deposition within 100–150 m of the highway (Sabin et al., Atmos Environ 2006;40:7528–7538).
One thing’s for sure, though. The fact that the investigators didn’t see even a hint of a correlation between autism and a similar measure to proximity to freeways makes me wonder very strongly whether this is simply nothing more than a finding that popped up in multiple comparisons due to random chance alone.
Let’s, for argument’s sake, however, assume that this isn’t the case. Let’s assume that the data are the data, and the correlation found appears not to be due random chance alone, at least using the data and techniques used in this study. I stil see a big problem: a confounding factor that wasn’t controlled. for. True, the authors did try to control for all the usual confounders in autism studies: child sex, ethnicity, level of education in the home, maternal smoking, maternal age, and preterm delivery. (One notes that they didn’t control for paternal age, which, as recent studies have shown, appears to correlate with an increased risk of autism.) But what did they fail to control for? If you remember my deconstruction of Dr. Raymond F. Palmer’s study that claimed to link proximity to mercury-emitting power plants to an increased risk for autism. Do you remember what that was? No reason to expect that you should, given that my post is over two and a half years old, but the criticism of Palmer’s study applies equally well here. Moreover, Joseph explicitly listed a number of complaints about Palmer’s study and demonstrated that the correlation that Palmer found disappeared once population density was accounted for. Why is this important? Urban and suburban regions with much denser populations tend to have more screening and intervention programs for autism and thus a higher rate of diagnosis. If we take a look at the distribution of CHARGE study subjects from the Sacramento Valley catchment area, it can be seen that they are scattered throughout central California. I couldn’t find an equivalent map for the Los Angeles basin catchment area.
The reason why controlling for population density and urbanicity is so important was well explained by Thomas A. Lewandowski in criticizing the Palmer study:
Lastly, the authors found that the most important determining factor for autism prevalence in their study was whether the child lived in an urban, suburban, or rural area. For example, residence in an urban school district resulted in a 473% higher rate of autism compared to rural districts. Similar findings have been reported by others (e.g., Deb and Prasad, 1994). The urbanization effect is nearly 8 times stronger than the effect suggested for mercury but is given relatively little discussion and is not even noted in the abstract. Since levels of many pollutants (including mercury) would be strongly correlated with urbanization/industrialization, this also leads one to question the mercury-autism association the authors report…Certainly a host of environmental and social variables associated with urbanization could be investigated as possible factors in autism. Alternatively, an increased tendency for diagnosis in urban localities could explain at least part of the increased incidence.
Urbanicity is also associated with proximity to Interstate freeways, although, to be fair, it is also associated with proximity to major roads. The denser the population, the more freeways and roads.
Another question that came up as I read this study was a simple one: How many total associations were the authors looking for in their data? Remember, the CHARGE study is looking for environmental factors that modulate autism risk, as described in this review article about the study published in 2006, coauthored by one of the study under discussion:
To structure the search for etiologic factors, we are beginning with known neurodevelopmental toxicants and hints from the immunologic evidence. Additionally, physiologic differences that might provide clues about susceptibility and mechanisms are being examined through characterization of metabolic, immunologic, and gene expression profiles, as well as genetic polymorphisms. Figure 1 shows five broad classes of exposures of interest: pesticides, metals, persistent pollutants with known or suspected neurodevelopmental or immunologic toxicity, medications and other treatments, and infections. Exposures from both the prenatal and early childhood periods are being investigated, with data primarily from three sources: a) extensive interviews with parents; b) laboratory analysis of xenobiotics in blood, urine, and hair specimens; and c) prenatal, labor and delivery, neonatal, and pediatric medical records.
This is represented in a figure:
In other words, the current paper is a report of a subset of analyses from a much bigger study.
The bottom line
Given that this published study is part of a much larger study looking at a lot of factors, there’s always the danger that the reported finding is nothing more than an anomalous result that can be attributed to random chance alone. Such findings are quite common when multiple comparisons are made; indeed, as the number of comparisons rises, so does the chance of finding a fluke correlation that mathematically is statistically significant but is in reality merely random chance in action. In other words, if investigators were looking for 100 different associations and found this one, it would be a different thing than if they were looking for only five and found this one. Without knowing the number of associations authors were looking for at the beginning without having to go to various other papers describing the study, it’s hard to judge whether this one correlation (autism prevalence with proximity to freeways either at birth or during the third trimester, which, let’s face it, correlate closely with each other, given that most expectant mothers don’t want to change residences when they are near the end of their pregnancy) means anything at all. My guess is that it probably does not; I would find the data more convincing if there were included actual measurements of pollutants compared to proximity to freeways and major roads, along with an analysis that shows that pollutants up to 309 m away from a freeway are still much higher than pollutants a mere 42 m from a major road. It would also have been a lot more convincing if there were a geographic effect consistent with the direction that the prevailing winds blow.
Epidemiological studies of clusters of conditions or diseases are fraught with false positives, so much so that there is even a name for the particular fallacy that all too many investigators fall prey to when looking for such clusters: The Texas sharpshooter fallacy. The fallacy is named for the apocryphal Texas sharpshooter who fired a bunch of bullets into the side of a barn and then drew a circle around the area where most of the bullets hit, calling that circle the bullseye. Similarly, epidemiologists look at the geographical location of subjects with a particular condition or disease, and then try to “draw a circle” around them that fits in with what they are looking for. If that’s what happened here, then the CHARGE investigators, believing that pollution might be a contributor to autism, tried to “draw the circle” near freeways and major roads, found one statistically significant result, and then concluded that proximity to freeways is a risk factor for autism, the implication of which (to the investigators) is that pollution is a predisposing factor to autism. I can’t help but note in looking at these results that it is is already known from a study earlier this year that in California autism cases tend to cluster in areas with high populations of more affluent, educated people, who, not surprisingly, tend to live in more affluent communities with good school services, special education, and autism screening and intervention programs.
This approach of examining geographic clusters is nearly always fallacious because (1) the cluster is usually the result of random chance or (2) there may be other reasons for the clustering. Personally, I suspect option #2, with the other reason for the clustering being the unaccounted-for confounding factor of urbanicity. In all fairness, the authors do mention in the discussion that their result might be due to “random chance or bias.” Also, in various interviews, Volk did say that their work needs to be replicated (although not, as far as I saw, that their results are likely to be due to random chance or an uncontrolled-for confounding factor). However, missing from most news reports was a level of skepticism about a study finding in essence only one barely statistically significant result among several.
One can’t help conclude that, if this association were true, then, as AutismNewsBeat pointed out, we would expect autism prevalence to have started its upward trend back in the late 1950s, after the passage of the Interstate Highway Act and the subsequent explosion of construction of interstate freeways. As we all know, however, this explosion didn’t actually begin until the early 1990s, which is the reason why anti-vaccine groups seized upon vaccines as the One True Cause of autism. Such are the hazards of confusing correlation with causation. I don’t think that that’s what the authors are necessarily doing here, but I do think they are doing a disservice to the public by not adequately trying to squash the misunderstandings about studies of epidemiological clusters.