The internet is a fabulous resource of information. It is one of those technological innovations for which you soon can no longer imagine how you lived without it. I certainly cannot imagine a project like science-based medicine prior to the web.
The web, however, is also a tremendous source of misinformation, opinion, and ideology. Also the volume of information, good and bad, can be overwhelming. We therefore are frequently asked the meta-question of how we conduct our research into specific topics, or how can the average layperson do their own research online.
Efficiently and effectively researching a complex topic is complex. It is a skill that needs to be developed, and it is especially difficult without having detailed knowledge of the specific topic ahead of time. Therefore there is no simple answer to this question, but I can offer some tips.
There are two main resources I use when searching a topic, Google and PubMed. They each have their strengths and weaknesses. For the average user, Google (or whichever general search engine you prefer) is likely going to be your first stop.
Using Google
When I am researching an unfamiliar topic the first thing to pay attention to is the search terms. You may not be using the proper or most precise terms in your search because you are unfamiliar with the jargon. You may need to keep searching with variations of descriptions of the topic in which you are interested. If you know the name of someone or an institution attached to that topic, include that in the search.
When you hit an article that refers to or is related to the topic, pay close attention to the specific terms that are used. They will become your new search terms as you hone in your search. Sometimes it take me several rounds of doing this before I get to the precise terms that get me to the content I want.
The strength of Google is its search capability. The weakness is that the results are going to be contaminated with many low-quality sites from an SBM perspective – meaning sites that are commercial or ideological. Once you find your optimal search terms, adding “skeptic” or “skeptical” to the search may bring you to the critical analysis for which you are looking.
Using PubMed
PubMed is a publicly available database of peer-reviewed journals in the medical field, including basic science and psychology. Overall this is a better resource than Google, but you need a higher level of familiarity with the topic. Often I use Google just to find the proper search terms to use in a PubMed search.
The strengths of PubMed include the fact that you are taken only to peer-reviewed primary scientific sources. The search engine, however, is not as robust or user-friendly as Google, so you really do need to know the technical search terms. You can also, however, search by the name of the publishing author or the journal, if you have that information.
An imprecise search can result in tens of thousands of results. You may have to wade through dozens of articles to find one that is relevant to your topic, but once you hit one relevant article that will likely lead you to the articles you want through better searches. Also, with PubMed, when you click on an article the site will also list other related articles. Look through those, and when you click on them you will get more related articles.
PubMed displays the abstracts of articles only. Abstracts are useful summaries, and may be adequate for your purposes – if all you want is to learn the bottom line. But for any detailed analysis you need to read the full article, as abstracts are often misleading. PubMed offers links to full articles when available, which is increasingly the case, but often the full article is not available online or is behind a paywall (an expensive paywall).
A well-written article will often contain an introduction which provides the authors’ summary of the topic, including an overview of current research. This is often an excellent way to understand the current thinking of researchers about that topic.
The most useful articles on PubMed are often going to be systematic reviews. If you want to go straight to these articles, then just include the word “review” in your search. Reading the three or four most recent systematic reviews of a specific topic or question is the quickest way to get a useful bottom-line summary of the status of the evidence.
Evaluating Resources
Not all resources are equally valid or useful. When it comes to published primary research, the reputation of the journal and the authors are the most important criteria. That is why I will search on the author names (in both Google and PubMed) to find out what other research they have done, or if they are involved in any ideological advocacy.
Luckily journal names often give away their focus and impact. Journals with names like “Nature” and “Science” have a long history and solid reputation. Journals with names like, “The Filipino Journal of Cancer Homeopathy,” likely don’t have the same reputation. But if in doubt, search for other articles in the same journal to see what they are all about.
When evaluating sites on the web, there are a few useful rules of thumb. Group sites are more reliable than sites dedicated to the work of a single individual. University sites tend to be of higher quality than commercial sites. Be suspicious of sites that are trying to sell you something, or that have a clear ideological ax to grind.
Otherwise, the general principles of skeptical analysis apply. Be wary of sites that are promoting conspiracy or fear mongering, or that are making claims that seem too good to be true.
In addition to SBM, another deep resource of medical information on the web is QuackWatch. On the homepage there is a section for “Nonrecommended Sources of Health Advice.” That is an excellent list of sources to avoid, unless of course you are looking for claims to debunk. There is also a list of recommended links.
Putting it all together
Perhaps the most difficult question when researching a specific topic is this – when are you done? When have you found all the relevant information? You may feel as if you have found many valuable resources, but perhaps you are missing that one article that changes everything. Again, there is no simple answer to this. The more time you invest in research, the more confident you can be that you have found all relevant information.
The only way to be sure is to do a systematic search, but that is frankly not feasible for the average person. Even for experts, like the authors of SBM, we rely upon other researchers to do systematic reviews, which can take hundreds of hours.
But, following the tips above, if you are reasonably diligent, you can wrap your head around a narrow topic or specific question with a few hours of online research.
It is always useful, however, to simply ask another person for their opinion, especially if they have more expertise in that area. I will often ask other experts – this is what I have found, am I missing anything?
One of the advantages of blogging is that it serves as a means of crowdsourcing. If, after doing my research, I did miss something, it is likely to pop up in the comments. The only way to be really sure you thoroughly covered a topic is to discuss it with others who have knowledge in the area. You have to engage with the community. That is part of the reason why scientists go to conferences and meetings.
If you are not a blogger, then simply engaging in the comments of a blog like SBM can serve the same purpose. You can also ask questions on forums and message boards. Of course, such online communities are likely to have their own culture and biases. You can easily get sucked into an echo chamber with an extreme point of view. Keeping your sources diverse is important here as well.
In truth the process of researching a topic is never truly done. Even for topics that I feel I have mastered, I am frequently encountering new viewpoints or nuances. Like science itself, it is an endless self-corrective process that requires you to be open to revision.
