to reflect how people start a sentence
when formulating their opinions.
StopFake is a relatively less-known
service. We thus selected a broader search string—“StopFake”—to
be able to collect enough relevant
opinions. The searches returned a
data corpus of 1,741 posts over six
months—October 2014 to March
2015—as in Figure 3. By “posts,” we
mean written contributions by individual users. To create a sufficient
dataset for analysis, we removed all
duplicates, including a small number
of non-relevant posts lacking personal opinions about fact checkers. This
filtering process resulted in a dataset
of 595 posts.
We then performed content analysis, coding all posts to identify and
investigate patterns within the data1
and reveal the perceptions users express in social media about the three
fact-checking services we investigated. We analyzed their perceptions of
the usefulness of fact-checking services through a usefulness construct
similar to the one used by Tsakonas
et al. 14 “Usefulness” concerns the extent the service is perceived as beneficial when doing a specific fact-checking task, often illustrated by positive
recommendations and characterizations (such as the service is “good”
or “great”). Following Mayer et al.’s
theoretical framework, 7 we categorized trustworthiness according to
the perceived ability, benevolence,
and integrity of the services. “Ability”
concerns the extent a service is perceived as having available the needed
skills and expertise, as well as being
reputable and well regarded. “
Benevolence” refers to the extent a service
is perceived as intending to do good,
beyond what would be expected from
an egocentric motive. “Integrity” targets the extent a service is generally
viewed as adhering to an acceptable
set of principles, in particular being
independent, unbiased, and fair.
Since we found posts typically re-
flect rather polarized perceptions of
the studied services, we also grouped
the codes manually according to sen-
timent, positive or negative. Some
posts described the services in a plain
and objective manner. We thus coded
them using a positive sentiment (see
To collect opinions about social
media user perceptions of Snopes
and FactCheck.org, we applied the
search term “[service name] is,” as
in “Snopes is,” “ FactCheck.org is,”
and “FactCheck is.” We intended it
as https://wordpress.com/), discus-
sion forums (such as https://offtopic.
com/), and online newspapers (such
com/) requested by Meltwater cus-
tomers, thus representing a large,
though convenient, sample. It col-
lects various amounts of data from
each platform; for example, it crawls
all posts on Twitter but only the Face-
book pages with 3,500 likes or groups
Figure 4. Positive and negative posts related to trustworthiness and usefulness per
fact-checking service (in %); “other” refers to posts not relevant for the research
categories (N = 595 posts).
0% 20% 40% 60% 80% 100%
Snopes (n = 385) FactCheck.org (n = 80) StopFake (n = 130)
Table 2. Snopes and themes we analyzed (n = 385).
Theme Sentiment Example
Positive (21%) Snopes is a wonderful Website for verifying things seen online; it is at
least a starting point for research.
Negative (10%) Snopes is a joke. Look at its Boston bombing debunking failing to
debunk the worst hoax ever ...
Positive (6%) […] Snopes is a respectable source for debunking wives’ tales, urban
legends, even medical myths ...
Negative (24%) Heh ... Snopes is a man and a woman with no investigative
background or credentials who form their opinions solely on Internet
research; they don’t interview anyone. […]
Positive (0%) No posts
Negative (21%) You show your Ignorance by using Snopes … Snopes is a NWO
Disinformation System designed to fool the Masses ... SORRY. I
Believe NOTHING from Snopes. Snopes is a Disinformation vehicle
of the Elitist NWO Globalists. Believe NOTHING from them ... […]
Positive (2%) Snopes is a standard, rather dull fact-checking site, nailing right and
left equally. […]
Negative (44%) Snopes is a leftist outlet supported with money from George Soros.
Whatever Snopes says I take with a grain of salt ...