When Scientific American first announced that they would publish Scientific American Mind, I hurried to subscribe, thinking it would keep me informed about new developments in a field I am passionately interested in. I have enjoyed the magazine, particularly the regular columns, the news items about research findings, the reviews that alert me to books I will want to read, the “Ask the Brains” Q and A, the challenging “Head Games” quiz, and the presentation of many intriguing ideas. The board of advisers is impressive, and the columns by Christof Koch, Scott Lilienfeld, Hal Arkowitz, the Ramachandrans and others have been consistently excellent. Unfortunately, some of the other articles have descended into pop psychology, speculation, poor science and even pseudoscience. Contributing editor Robert Epstein’s articles have particularly raised my blood pressure.
Love-Building Exercises
In December 2009 I was annoyed enough to write this letter to the editor:
After reading Robert Epstein’s article in the last issue, I had to go back to the cover and verify that the word “scientific” was indeed part of the title of your magazine. The Love Building Exercises he recommends are more appropriate to a magazine of fantasy and science fiction.
Two as One — feeling that the two of you have merged?
Soul Gazing — looking into the very core of your beings?
A Mind-Reading Game — wordlessly trying to broadcast a thought to another person?
Love Aura — feeling “eerie kinds of sparks” when your palm is close to another’s?Thought transfer? Auras? Come on! Shame on you for publishing such metaphysical pseudoscientific psychobabble!
They published my letter to the editor with the heading “Hating ‘Love’.” There was no response from the author.
Are You Mentally Healthy?
In a March, 2010, article, “Are You Mentally Healthy?” Epstein presented a screening test that he had developed for mental health disorders and named after himself. He thought his test was more reliable than any of the other tests he found on the Internet because those other tests had not been scientifically validated. His “validation” consisted of his own findings that scores on his test predicted seven important factors related to mental health, such as whether they were employed, how highly they rated their personal and professional success, and whether they had ever been in therapy. (John Nash had been treated for his schizophrenia, but he was employed, won a Nobel prize, and had lasting personal relationships. One wonders how he would have done on Epstein’s test.) Essentially Epstein tried to defend one unvalidated test by showing that it correlates with another unvalidated list of factors.
For fun, I took Epstein’s test. I could see that most of the questions were designed to elicit specific symptoms of depression, mania, anxiety, compulsions, etc. and it appeared to be little more than part of a checklist that a psychiatrist might use to remind him of questions to ask in taking a conventional psychiatric history.
Epstein has not tested people who are known to have mental illness and people who are known to be mentally healthy, but only random people who found his questionnaire on the Internet. He has not defined mental health, much less measured it. He has only shown that his test scores predict a person’s answers to specific questions that are part of the test itself, questions that he personally thinks are related to mental health. People with mental illnesses may not answer the way he thinks that they will. And on the other hand, mentally healthy people might answer the way he thinks only mentally ill people would. The only way to be sure that a survey works is to “test the test”: to see if mentally ill people actually score high on the test. Epstein hasn’t done that.
Any test, questionnaire or instrument must be checked for both reliability and validity: reliability means it will give consistent, reproducible results, and validity means it has been compared to some other standard to ensure that it is actually measuring what it claims to measure. For instance, a new kind of thermometer might reliably give the same result every time, but the readings wouldn’t be valid for diagnosing a fever unless they agreed with the readings on a mercury thermometer. A stopped watch reliably shows the same time each time it is consulted, but it is not valid for telling the time. Epstein may think he has “validated” his questionnaire, but he hasn’t.” He has not shown that the test has any validity for predicting the presence of mental illness.
I tried checking none of the items, and it told me
You haven’t checked off any items, which suggests that your mental health is excellent. If you still have concerns, you can find qualified, licensed counselors and therapists at websites such as Find-a-Therapist.com, GoodTherapy.org, NetworkTherapy.com, PsychologyToday.com, Metanoia.org, AskTheInternetTherapist.com, and TherapistLocator.net. You can also get referrals through your family physician, HMO, or local hospital or clinic. If you are worried that you are losing control of your life, consider taking the test at HowInfantilizedAreYou.com.
Then I tried checking all of the items. This time it told me
In some respects [sic] you scored outside the range of functioning that is usually considered normal. This suggests that you should probably [sic] consult with a qualified mental health professional for further testing or treatment. Area(s) of possible concern (expressed in the diagnostic language that will be familiar to your therapist):
Substance Abuse
Psychosis
Depression
Mania
Bipolar Disorder
Mood Disorder
Phobia
Social Phobia
Obsessive-Compulsive Disorder
Posttraumatic Stress Disorder
Generalized Anxiety Disorder
Anxiety Disorder
Relational Disorder
Sexual Disorder
Eating Disorder
Impulse Disorder
Personality Disorder
Somatoform DisorderYou can find qualified, licensed counselors and therapists at websites such as Find-a-Therapist.com, GoodTherapy.org, NetworkTherapy.com, PsychologyToday.com, Metanoia.org, AskTheInternetTherapist.com, and TherapistLocator.net. You can also get referrals through your family physician, HMO, or local hospital or clinic. If you’re worried that you’re losing control of your life, consider taking the test at HowInfantilizedAreYou.com.
Then I tried checking all the odd-numbered items, resulting in a verdict of pretty healthy but having a possible eating disorder, followed by the same canned universal suggestions.
Checking all the even-numbered items gave me a possible diagnosis of social phobia. Just for the halibut, that time I also reported that I was 10 years old and had a doctorate. That kind of question doesn’t go into the scoring, but Dr. Epstein uses it for his research-by-Internet-stealth. Underlining just how unreliable such research is.
How Do You Handle Stress?
Epstein’s latest article is another example unworthy of Scientific American. In the September/October 2011 issue he gives us “Fight the Frazzled Mind: A new study suggests that preventive, proactive approaches are the most helpful — and that our stress management IQ is painfully low.” The new study is one Epstein did himself and presented at a conference but did not publish in a peer-reviewed journal.
The study looked at 3304 subjects who completed an online test. They were asked to rate, on a 10-point scale, how stressed they were, how generally happy they were, and how much success they had had in both their personal and professional lives. The main body of the test involved questions in four areas of competency: manages sources of stress, practices relaxation techniques, manages thoughts, and prevents stress from occurring. As far as I can see, this identification of four competencies and the corresponding questions are nothing but his own invention. Some examples of individual questions that he thinks can be used to measure those competencies:
- I try to schedule appointments and meetings so that they won’t overlap.
- I schedule some relaxation time every day
- I’m aware that my thinking is sometimes unclear or irrational
- I keep an up-to-date list of things I’m supposed to do.
He says he was surprised by one of his findings: that prevention is by far the most helpful competency when it comes to managing stress. His take-away message is that it is better to avoid stress in the first place than to use techniques like relaxation after stress has developed. I am surprised that he finds this surprising.
He says his study also shows that people who have had training in stress management are better at it, and the greater number of hours of training, the better the skills. He doesn’t explain, elaborate, or quantify. “Stress management training” is not defined.
Then, out of the blue, he offers six strategies he says were “suggested by the new study” to fight stress before it starts:
- Seek [stressors] and kill
- Commit to the positive
- Be your own personal secretary
- Immunize yourself (through exercise, thought management and relaxation techniques)
- Make a little plan (for each day)
- And make a big plan (for the long term)
But these weren’t really suggested by the study, they were strategies that he had already decided ought to be stress-reducing and therefore ought to be included in his questionnaire. Do I smell circular reasoning? And of course, he has no evidence that efforts to adopt these strategies will have any measurable effect.
Then he says the worst news is that on his 100 point scale, people scored an average of 55.3.
If you think of that as a score on an exam at school, that means that on average, people get a grade of F when it comes to managing the inevitable stress they face in their lives.
A Faulty Method
This seems to be Epstein’s modus operandi: he thinks up his own questionnaire to try to measure something, and without even trying to validate it he proceeds to use it in a study, and then gives talks and writes popular articles about his results and gives pop psychology advice allegedly based on the studies. But these studies never get published in peer-reviewed journals and never show up in PubMed. Epstein’s website offers the stress questionnaire and a book of stress relief games, along with tests he has developed to measure such things as “adultness” and “love competency.” No, I’m not making this up! I took the adultness test and found it very entertaining. You might too. His questionnaires are reminiscent of the kind of questionnaires that are ubiquitous in popular magazines, where your score allegedly predicts whether your marriage is likely to last or tells you whether your self-esteem is high or low.
The flaws of this method are obvious, as can be seen in the new “Stress” article. Questionnaires must be validated before they can be used to measure anything. Terms like “stress” must be objectively defined. Self-reports of stress, happiness and success may not correspond reliably to any quantifiable reality. Subjects who self-report as happy, successful and non-stressed can be expected to answer the questionnaire items from the biased perspective of their self-image. And a score on a made-up test can hardly be compared to an F grade in school.
Conclusion
Epstein is much better at self-promotion than at science. In my opinion, Scientific American Mind would be better off without him. Let this stand as an open letter recommending that they remove him from the position of contributing editor and that if they consider publishing any more of his articles, they first submit them to peer review by rigorously scientific psychologists with good critical thinking skills, such as Scott Lilienfeld, who is already on their board of advisers and is also a fellow of the Committee for Skeptical Inquiry and the editor of The Scientific Review of Mental Health Practice. I’m picking on Epstein as a bad example (and a particularly prolific one), but he is not the only offender. Other similarly questionable articles have slipped past the editors. With a little weeding, Scientific American Mind could be what its name promises: scientific.