Appreciating the value of top rankings, Webmasters have learned to
optimize their pages so big search
engines rank them more highly. This
has spawned a worldwide industry of
search-engine-optimization consultants, whose techniques are grouped
into two categories: white-hat, ensuring that search engines easily analyze
a site and are accepted by search engines; and black-hat, including hidden text, as in white text on a white
background, considered illicit by
most search engines and upon discovery generally punished with degraded
ranking.
Search engines clearly have a le-
gitimate interest in fighting inappro-
priate third-party optimization tech-
niques to ensure their search-result
quality; for instance, sites with no
other purpose than linking to specific
sites to increase page rank (link farms)
are black-hat and must be dealt with
accordingly, though punishment can
be problematic for multiple reasons:
First, sudden ranking demotion
and resulting diminished inflow of vis-
itors have major effects on businesses,
as illustrated by the cases of Skyfacet
(which reportedly lost $500,000 of rev-
enue in 2006) and MySolitaire (which
reportedly lost $250,000 the same
year14). Only a few cases, including the
companies SearchKing18 and Kinder-
start11 involving lawsuits over page
figure 1. how ranking works.
spider/Crawler
searches Web,
storing pages in index
index
search engine
software
Algorithm accesses index,
determining most relevant
results for queries based on:
˲ ˲ keyword analysis (such as
word location and frequency)
˲ ˲ link analysis (such as number
of links from other pages and
rank of these pages)
rankings and German car manufacturer BMW, received notable media attention. Many more cases of dropped
ranking have been condemned to virtual silence, among them search-engine optimizer Bigmouthmedia.
Second, though market-leader
Google has published guidelines on
creating “Google-friendly” pages, the
line between permitted and illicit
practices is blurry at best. 20 For example, Google’s guidelines rightly warn
against cloaking, “the practice of presenting different content […] to users
and search engines.” 13 However, to a
certain extent cloaking can be justified
and used with good intent by major
sites without penalty. 8 For instance,
the Wall Street Journal uses it to show
full versions of pay-per-view articles to
Google’s indexing program. 8
The difficulty of straddling the line
between permitted and illicit practices is further illustrated by a case
involving paid links: In February 2009
Google punished its subsidiary Google
Japan through a page rank demotion
for paying for online reviews of a new
widget. 23 While Google’s attempt to
play by its own rules is positive the
case highlights the difficulty of distinguishing permitted from illicit optimization techniques. A leading U.S.
commentator in online search asked:
“If Google itself […] found itself in
this situation, how are ordinary Web
Web
user
makes request
sites to be expected to know the ‘rules’
about what they can or cannot do?” 23
Third, our research supports the
idea that there is no established pro-
cess of announcement or appeal prior
to rank demotion. Companies af-
fected usually realize their fate only
through a sudden loss of traffic or rev-
enue. In a personal interview [2008],
the CEO of an educational company
told us: “The office called me and told
me [...] that revenue was down [...], so
I checked our logs and our history [...]
It was all on one day. We were up to 14
million pageviews per month, and on
one day it dropped 70% and [stayed
there], and that was it.”
Fourth, options are limited for
companies affected by ranking demo-
tion. One interviewee recalls his com-
pany got no response, even though he
personally went to the search engine
firm’s headquarters for assistance.
Fifth, several allegations in the
blogosphere claim large players are
treated better than their less-powerful
counterparts. For example, in May
2008, Hewlett-Packard began offering free blog templates, including
hidden links to its own pages, a quick
way to gather “high-quality” links and
clicks. 25 However, there was no evidence of punishment by major search
engines, sparking significant controversy in the community. 25
Finally, search engines have punished Web sites using search-engine
optimization, as well as the search-engine-optimization companies themselves. In the case of SearchKing, a
search-engine consulting company
in Oklahoma, U.S. courts have found
that Google “knowingly and intentionally” dropped the company’s Web
sites in its rankings to punish what it
deemed illicit ranking manipulation
that SearchKing had carried out for its
clients. 18
Rationale for Regulation
Several researchers have pointed to
the dangers of targeted manipulation,
arguing it undermines values like free
speech, fairness, economic efficiency,
and autonomy, as well as the institution of democracy. Concerning democracy and free speech, Introna and
Nissenbaum17 argued a decade ago
that search engines’ broad structural
bias can lead to underrepresentation