Appreciating the value of top rankings, Webmasters have learned to optimize their pages so big search engines rank them more highly. This has spawned a worldwide industry of search-engine-optimization consultants, whose techniques are grouped into two categories: white-hat, ensuring that search engines easily analyze a site and are accepted by search engines; and black-hat, including hidden text, as in white text on a white background, considered illicit by most search engines and upon discovery generally punished with degraded ranking.

Search engines clearly have a le-
gitimate interest in fighting inappro-
priate third-party optimization tech-
niques to ensure their search-result
quality; for instance, sites with no
other purpose than linking to specific
sites to increase page rank (link farms)
are black-hat and must be dealt with
accordingly, though punishment can
be problematic for multiple reasons:
First, sudden ranking demotion
and resulting diminished inflow of vis-
itors have major effects on businesses,
as illustrated by the cases of Skyfacet
(which reportedly lost $500,000 of rev-
enue in 2006) and MySolitaire (which
reportedly lost $250,000 the same
year14). Only a few cases, including the
companies SearchKing18 and Kinder-
start11 involving lawsuits over page

figure 1. how ranking works.

spider/Crawler searches Web, storing pages in index

index
search engine
software
Algorithm accesses index,
determining most relevant
results for queries based on:
˲ ˲ keyword analysis (such as
word location and frequency)
˲ ˲ link analysis (such as number
of links from other pages and
rank of these pages)

rankings and German car manufacturer BMW, received notable media attention. Many more cases of dropped ranking have been condemned to virtual silence, among them search-engine optimizer Bigmouthmedia.

Second, though market-leader Google has published guidelines on creating “Google-friendly” pages, the line between permitted and illicit practices is blurry at best. 20 For example, Google’s guidelines rightly warn against cloaking, “the practice of presenting different content […] to users and search engines.” 13 However, to a certain extent cloaking can be justified and used with good intent by major sites without penalty. 8 For instance, the Wall Street Journal uses it to show full versions of pay-per-view articles to Google’s indexing program. 8

The difficulty of straddling the line between permitted and illicit practices is further illustrated by a case involving paid links: In February 2009 Google punished its subsidiary Google Japan through a page rank demotion for paying for online reviews of a new widget. 23 While Google’s attempt to play by its own rules is positive the case highlights the difficulty of distinguishing permitted from illicit optimization techniques. A leading U.S. commentator in online search asked: “If Google itself […] found itself in this situation, how are ordinary Web

 

Web

user makes request

sites to be expected to know the ‘rules’ about what they can or cannot do?” 23

Third, our research supports the
idea that there is no established pro-
cess of announcement or appeal prior
to rank demotion. Companies af-
fected usually realize their fate only
through a sudden loss of traffic or rev-
enue. In a personal interview [2008],
the CEO of an educational company
told us: “The office called me and told
me [...] that revenue was down [...], so
I checked our logs and our history [...]
It was all on one day. We were up to 14
million pageviews per month, and on
one day it dropped 70% and [stayed
there], and that was it.”
Fourth, options are limited for
companies affected by ranking demo-
tion. One interviewee recalls his com-
pany got no response, even though he
personally went to the search engine
firm’s headquarters for assistance.

Fifth, several allegations in the blogosphere claim large players are treated better than their less-powerful counterparts. For example, in May 2008, Hewlett-Packard began offering free blog templates, including hidden links to its own pages, a quick way to gather “high-quality” links and clicks. 25 However, there was no evidence of punishment by major search engines, sparking significant controversy in the community. 25

Finally, search engines have punished Web sites using search-engine optimization, as well as the search-engine-optimization companies themselves. In the case of SearchKing, a search-engine consulting company in Oklahoma, U.S. courts have found that Google “knowingly and intentionally” dropped the company’s Web sites in its rankings to punish what it deemed illicit ranking manipulation that SearchKing had carried out for its clients. 18

 

Rationale for Regulation Several researchers have pointed to the dangers of targeted manipulation, arguing it undermines values like free speech, fairness, economic efficiency, and autonomy, as well as the institution of democracy. Concerning democracy and free speech, Introna and Nissenbaum17 argued a decade ago that search engines’ broad structural bias can lead to underrepresentation