4 Chapter 4 (Non-academic Sources)
There are some sources that a simple search engine will pull up time and again. Most of these sources are poorly vetted or shallow at best or have an agenda or perpetuate errors at worst. However, instead of summarily dismissing them, it is worth diving into sources that some people might be inclined to lean on at first and then exploring why they are problematic.
References
The best of the worst, references are works that contain facts and information that one can use to “look up” basic information. Most students in K-12 schools are familiar with using references at least to some degree, and they can feel like a comfortable place to start. However, references are a great example of how a “source” can be useful for general interest inquiries but not all that useful for actual research. References are actually not sources in any pure sense. Britannica and Wikipedia do not actually perform research. Instead, they convey the results of the research that others hand conducted, at best.
This does not mean that references are useless. It means that what references can be used for it limited. More often than not, references convey information that is compiled and summarized by a non-expert, using more credible sources at their base. In essence, most encyclopedia entries are far closer in quality to college student papers than they are to expert works.
Consider dictionaries, a staple of the K-12 essay. Even dictionaries are wrong, or at least imprecise, based on editorial practices. For example, as of January 2022, the Merriam-Webster dictionary offered the following definition of carnosaur: “any of a group (Carnosauria) of very large theropod dinosaurs (such as a tyrannosaur)”. The problem with this definition is that tyrannosaurs are not carnosaurs, they are coelurosaurs (a related family that is still a different group), and this has been pretty well-established since the 1990s. Does this make a huge difference? It does if the researcher finds information specific to carnosaurs or tyrannosaurs and mistakenly applies it across both groups.
The same dictionary’s definition of autism also encounters some problems: “a variable developmental disorder that appears by age three and is characterized especially by difficulties in forming and maintaining social relationships, by impairment of the ability to communicate verbally or nonverbally, and by repetitive behavior patterns and restricted interests and activities.” Note the language “appears by age three” and consider what that means. ASD is sometimes not diagnosed until adulthood (some 50-year-olds have received such a diagnosis for the first time), and so this definition can be problematic if the person using the definition over-applies it and assumes that the four words in question have medical significance or represent technically sound specificity.
Encyclopedias (online or otherwise) run into the same problems to a lesser degree. By default, when a non-expert tries to condense technical information, that information begins to lose accuracy. This does not mean that researchers should not use references. It means that they should not use references as sources of information, but rather as a means of better understanding information that has been researched.
List-icles and Opinion Pieces
A number of websites (e.g. ProCon.org) offer lists of ideas for why someone might be in favor of a particular proposition or against it. However, not only are these websites often engaging in the fallacy of a false dichotomy, they seldom actually provide evidence for their claims. Instead, they simply list claims made by individuals without providing the underlying evidence or warrants to support those claims. As a result, all they really establish as “sources” is that some people were able to think of the claims as ideas that might be reasons someone might have a belief. They do not, even, establish that anyone believes such things.
One of the most common mistakes made in college writing is when students mistake finding someone who agrees with their opinions with finding evidence. For example, I might find someone who agrees with me that the Loch Ness monster exists, but that does not mean I have any additional evidence of its existence. Instead, “evidence” is present when someone is able to provide both demonstrable facts in support of a claim and is able to articulate the way in which that evidence supports the claim.
For example, the fact that proto-feathers are present on multiple “raptor” dinosaurs and on non-carnivorous dinosaurs is evidence that feather-like structures likely evolved early in the dinosaur family tree (because it is more likely that the same trait developed once and then spread than that the same trait developed multiple times independently in related species). Note that this does not mean I’ve “won,” because convergent evolution is a real phenomenon. It is, however, evidence for my claim. By contrast, having made a friend on Reddit who agrees that raptors look “dumb” with feathers is not evidence.
Likewise, editorial and opinion pieces are frequently carried by newspapers and other organizations. However, that does not mean that they are reliable sources of information. In fact, opinion pieces are so unreliable that courts have actually found that they are not accountable for accurate presentation of facts. In one case, media personality Tucker Carlson’s own lawyers successfully argued in his favor that he could not be held accountable for statements he made because “any reasonable viewer arrives with an appropriate amount of skepticism about the statements he makes” despite the commentary itself being framed with the claim “remember the facts of the story. These are undisputed.”
The lesson for any researcher is clear—be skeptical of almost any supposed source, and do not assume that just because a source claims to be offering facts that it is, in fact, doing so, even if the source shares a platform with news agencies.
Think back to the Toulmin model discussed in an earlier chapter. If the article in question (be it a list, a blog, or an opinion piece) offers data, warrants, and claims then it might be a worthwhile place to start research and get ideas. However, usually each of these pieces is nothing more than another person’s opinion.
News Reports
Of course, the prior section leads to the issue of news reports themselves. There are often suspect—not because they are wrong, but because they have a very different function than presenting information a complete view of an issue. News outlets frequently run into the same problems as references, in that they are staffed not by experts in individual disciplines but rather by individuals who are simply trying to convey information quickly. Likewise, the pressure to offer a story as soon as possible can frequently lead to inaccuracies in the name of expediency.
More importantly than those structural issues, however, is a deeper problem with news outlets. If a paleontologist discovers a new fossil and reports it to the news, the news outlet does not actually perform an examination of the fossil. Instead, it conveys what the paleontologist says about the fossil. If a politician tries to support a new law, the news outlet does not perform a deep analysis of the implications of the law—it quotes what the politician has to say about the law. Sometimes, in the interest of parity, the news outlet will take an additional step to find a paleontologist who disagrees with the first or to find a political opponent who dislikes the law. This might create an illusion of doubt where it does not exist among experts, but it might also force the discussion into binaries when a range of options exist. Note that the word used for news is medium (the plural being media), a word that actually refers to the role that news plays—it’s a bridge between the actual source and the reader. It is only as accurate as the original source, and then only if the bridge itself conveys the information reliably.
Frequently, news articles run into the problem that their sources of information are people who are biased on the news being reported. They might learn about a new business venture from a mega corporation from a press release put out by that very same corporation. A mayor might announce a new city-wide project and discuss all the benefits without explaining the drawbacks. In these cases, the news outlet does not always engage in fact-checking. Frequently, it will simply say “so-and-so says the project will bring new jobs,” and then expect the reader to evaluate whether or not the mayor in question was being honest.
An additional complication that comes from news sources is that sometimes they face pressure to provide “fair” reporting of politically charged subjects. Often, this means finding individuals with differing viewpoints and giving both individuals equivalent time or space. People receiving these two seemingly balanced reports might conclude that there are equal chances of each perspective being valid. That is seldom the case. The seeming parity is an artifact of how the subject is presented, not how much evidence there is to support each claim.
Activist Websites and Private Projects
One of the most concerning issues with a web search is how easy it is to confuse quantity with quality, and how easy it is to be led astray. It is very easy to come across private websites promoting individual causes, or else works that exist to further one particular person’s viewpoint. Some of these websites might even look “official” or professionally done. That does not mean that the information on these websites is correct.
Consider the issue of vertebrate evolution and David Peters, a paleo artist with a scientifically trained background. A simple search engine check would show that he seems to have some credentials in the subject of evolution and he has multiple published books. Maybe his work should be fine for “just facts,” especially since his website can be on the first page of sources? No. While Peters updates his webpage actively (helping it move up in search engine algorithms) and while his work in the 1980s, 1990s, and even a little bit afterward tended to represent scientific consensus, at some point he started pushing a number of unsupported claims about animal evolution, and he did so despite the fact that multiple experts offered correction on his mistakes. Most of his claims were unsupported and obviously inaccurate. However, a researcher unaware of this background could be misled.
Note that the same steps that were introduced in previous chapters would be highly useful, here. Is there independent confirmation? No. Transparency? Not really.
On the subject of neurodiversity, the organization Autism Speaks is one of the largest organizations that addresses ASD. However, the organization has frequently been criticized by members of the ASD community, specifically for furthering negative portrayals of neurodiversity and for excluding voices of the ASD themselves. It has been steadily criticized for privileging the viewpoint of its founders and not taking into consideration the current state of research. Here, the popularity and the high profile of this organization can lead to other problems, because it is relatively easy to find other sources that have gained their information from Autism Speaks. Therefore, tracing each source back to its origin is essential to finding quality research.
Unfortunately, a number of students have learned to trust .org and .gov websites due to overly simplistic one-size-fits-all guidelines from their childhood. However, .orgs are often transparently partisan websites, representing politically motivated organizations that actively seek to spread their own version of information as part of a persuasive effort to convert others to their cause–PETA.org, NRA.org, youthrights.org, and so on. Additionally, .gov websites are by definition overseen by those in political office, at least indirectly, and it’s probably unrealistic to expect whitehouse.gov to offer a nuanced account of the agenda of the chief executive.
Similarly, videos might be persuasive (they engage senses and are accompanied by clear charts or graphics, or else other images). Those videos, however, are often produced with funding or resources of those with an agenda. They are then written and directed by individuals with personal opinions and investment in the subject. They are seldom backed up with clear references or objective analysis. All videos (and many activist documentaries) really represent are editorials turned into video form.
Consider one widely distributed documentary that was aimed at the fast food industry in the 2000s. Supposedly a chronicle of one person’s journey through eating fast food, it left out a number of details: the star and his romantic partner were both Vegans, which likely had an impact on his body’s reaction to the food involved, his psychological responses to the diet, and even his motives for the documentary; he had a prior history of a chronic health condition that was left out of the movie; and he forced himself to eat past the point of being full (which is not how most people eat). When challenged, the documentary maker refused to share his food logs, however, and no one else was able to match his results–either in terms of weight gain or side effects–using the same protocol. The documentary was unquestionably persuasive, but it does not actually provide any actual evidence. Instead, it represents a deliberate attempt to attack an industry. It being a successful attack does not increase the factual validity of its claims.