Reliable sources: An interview with Factcheck.org

I have a post up at The Atlantic Tech featuring an interview I did with Brooks Jackson, Director of Factcheck.org about determining reliable sources. Factcheck.org is a terrific resource, and Brooks’ insights are excellent. Please head over to The Atlantic and read the interview. Here’s a taste:

We tend to be more skeptical of assertions that run counter to our existing worldview. How can we adjust for this bias of “motivated skepticism“? In such situations, it seems our reasoning capabilities are coming to the service of our emotions, to ill effect. Is it ever the case that we ought to employ less critical thinking?

In unSpun, Kathleen Jamieson and I argue that to keep from being fooled by this common human tendency, its a good idea to keep asking yourself “Am I missing something? Does the other guy have a point here?” It also helps to be aware of this universal psychological tendency, and for teachers to point out examples of it.

Kathleen doesn’t like the term “critical thinking” because it implies to some that they should automatically be critical. We prefer “analytical thinking.” If you look at it that way, I think there’s no danger of being too analytical. I agree that there is a danger of automatically distrusting anything said by people in authority. In that sense, yes, there is a danger of too much “critical” thinking. It’s one thing to be skeptical, which is good. It’s another to be cynical, which is a sort of naive belief that everybody is lying.

For more on this from Factcheck.org, check out their Tools of the Trade:

A Process for Avoiding Deception

1. Keep an open mind. Most of us have biases, and we can easily fool ourselves if we don’t make a conscious effort to keep our minds open to new information. Psychologists have shown over and over again that humans naturally tend to accept any information that supports what they already believe, even if the information isn’t very reliable. And humans also naturally tend to reject information that conflicts with those beliefs, even if the information is solid. These predilections are powerful. Unless we make an active effort to listen to all sides we can become trapped into believing something that isn’t so, and won’t even know it.

2. Ask the right questions. Don’t accept claims at face value; test them by asking a few questions. Who is speaking, and where are they getting their information? How can I validate what they’re saying? What facts would prove this claim wrong? Does the evidence presented really back up what’s being said? If an ad says a product is “better,” for instance, what does that mean? Better than what?

3. Cross-check. Don’t rely on one source or one study, but look to see what others say. When two or three reliable sources independently report the same facts or conclusions, you can be more confident of them. But when two independent sources contradict each other, you know you need to dig more deeply to discover who’s right.

4. Consider the source. Not all sources are equal. As any CSI viewer knows, sometimes physical evidence is a better source than an eyewitness, whose memory can play tricks. And an eyewitness is more credible than somebody telling a story they heard from somebody else. By the same token, an Internet website that offers primary source material is more trustworthy than one that publishes information gained second- or third-hand. For example, official vote totals posted by a county clerk or state election board are more authoritative than election returns reported by a political blog or even a newspaper, which can be out of date or mistaken.

5. Weigh the evidence. Know the difference between random anecdotes and real scientific data from controlled studies. Know how to avoid common errors of reasoning, such as assuming that one thing causes another simply because the two happen one after the other. Does a rooster’s crowing cause the sun to rise? Only a rooster would think so.

Who Wikipedia trusts

Lots of digital ink has been spilled on the trustworthiness of Wikipedia, and the circumstances in which it’s appropriate to use it as a source. Much more interesting, in my view, is the opposite question: what sources does Wikipedia trust? In our age of Truthiness, sorting good information from bad may be more critical than ever. It’s for that reason that factcheckers seem to be making a comeback. So how, exactly, does Wikipedia manage that sorting process? Do they differentiate between The New York Times and the National Review? Does the Congressional Budget Office count as more reliable than the Heritage Foundation? Than the Brookings Institution?

To try and find out, I visited Wikipedia’s Identifying Reliable Sources page. And while it didn’t answer many of my questions, I gleaned several interesting nuggets about Wikipedians’ idea of reliability. For instance:

  • “In general, the more people engaged in checking facts, analyzing legal issues, and scrutinizing the writing, the more reliable the publication.”
  • “When available, academic and peer-reviewed publications, scholarly monographs, and textbooks are usually the most reliable sources.”
  • “Mainstream news sources are generally considered to be reliable. However, even the most reputable news outlets occasionally contain errors. Whether a specific news story is reliable for a specific fact or statement in a Wikipedia article is something that must be assessed on a case by case basis. When using news sources, care should be taken to distinguish opinion columns from news reporting.”
  • “The statement that all or most scientists or scholars hold a certain view requires reliable sourcing that directly says that all or most scientists or scholars hold that view. Otherwise, individual opinions should be identified as those of particular, named sources… Stated simply, any statement in Wikipedia that academic consensus exists on a topic must be sourced rather than being based on the opinion or assessment of editors.”
  • “Anyone can create a website or pay to have a book published, then claim to be an expert in a certain field. For that reason self-published media—whether books, newsletters, personal websites, open wikis, blogs, personal pages on social networking sites, Internet forum postings, or tweets—are largely not acceptable. This includes any website whose content is largely user-generated, including the Internet Movie Database, Cracked.com, CBDB.com, and so forth, with the exception of material on such sites that is labeled as originating from credentialed members of the sites’ editorial staff, rather than users.”

More than anything, I was struck by how conservative these guidelines are. Wikipedia wouldn’t trust itself, for instance, being a user-generated project. On the one hand, that should put many of its more traditionally-minded critics at ease. On the other, it offers few new ideas about reliability.

Wikipedia’s reliability guidelines raise as many questions as they answer. But I think it’s important, as we all struggle to determine what’s reliable and what’s not, to look to innovative and successful collaborative projects like Wikipedia for guidance. There may not be much new there, but it’s not a bad starting point for discussion.

One point of interest to me: not only do Wikipedians maintain significant skepticism towards the press, they specifically don’t go in for the classic reporter’s line “many economists think”.

Bonus: PolitiFact has a post outlining their fact checking system here. I wish they would have gone further in identifying how they deal with reliability of sources. Perhaps I’ll ask them more about it.