Society | DOI: 10.1145/1467247.1467254
Leah Hoffmann
crowd control
Using crowdsourcing applications, humans around the world are transcribing
audio files, conducting market research, and labeling data, for work or pleasure.
ThoUGh CoMpUteRS have
outstripped us in arithmetic and chess, there are still
plenty of areas where the
human mind excels, such
as visual cognition and language processing. And if one mind is good, as
the proverb goes, two—or two thousand—are often better. That insight,
and its consequences, drew worldwide
interest with the 2004 publication of
James Surowiecki’s best-selling The
Wisdom of Crowds, which argued that
a large group of people are superior at
certain types of rational tasks than individuals or even experts.
Now researchers are turning to
computers to help us take advantage
of our own cognitive abilities and
of the wisdom of crowds. Through a
distributed problem-solving process
variously known as crowdsourcing,
human computation, and computer-aided micro-consulting, answers are
solicited online to a set of simple, specific questions that computers can’t
solve. Is this a picture of a fish? Do
you like that style of shoe? How many
hotels are on St. George’s Island, and
which ones have Internet access?
The amateur, often anonymous
workers who agree to execute these
tasks are usually given some sort of
social or financial incentive. A few
cents might buy the answer to a simple
data-labeling task, while a more ardu-
ous job like audio transcription could
require a couple of dollars. Reposition
the task as a game, and many people
even “work” for free. Either way, the
possibilities—for creating corpuses of
annotated data, conducting market research, and more—have both computer scientists and companies excited.
One of the oldest commercial
crowdsourcing applications is Amazon’s Mechanical Turk. Named after
a famous 18 century chess-playing
th
“machine” that was secretly operated
by a human, it offers a flexible, Web-based platform for creating and publicizing tasks and distributing micro-payments. Since its launch in 2006,
Turk has spawned both a vocabulary
and a mini-marketplace. Workers,
or “Turkers” (there are more than
200,000 in 185,000 countries, according to Amazon), select “Human Intelligence Tasks” (HITs) that match their
interests and abilities. Motivations
vary. Some work odd hours or at night
to generate extra income, while others simply desire a more productive
way to kill time online, like solitaire
with financial rewards. As in the of-fline world, more money buys faster
results, and Amazon’s HIT requesters
often experiment to find a pay scale
that matches their needs.
Also part of the Turk economy are
companies like Dolores Labs and Cast-ingWords, which rely on Amazon’s
technology to power their own crowdsourcing applications. Dolores Labs,
based in San Francisco, posts Turk
HITs on behalf of its clients, then filters
the answers through custom-built software systems to check for quality and
generate meaningful results. Data is ultimately used to perform tasks like filter
comment spam, tag data for search engine optimization, and research market
trends.
“Many companies don’t have the
resources to describe tasks, put them
up online, and manage the data they
get,” explains Lukas Biewald, the
company’s founder and CEO. Nor do
they have time for Dolores’s extensive
quality-control measures, which include creating “test” questions whose
answers are already known, checking
responses against one another, tracking individual answer histories, and
creating a confidence measure with
which to weight the resulting data.
Dolores also guides clients through
the many variables that are involved
in designing a crowdsourced project.
How arduous is each task? How quickly are results needed? How would clients like to deal with the statistical
outliers that are caught by Dolores’
quality-control algorithms? If you’re
checking user-generated content for
pornography, for example, you might
err on the side of caution.
According to Biewald’s estimates,
some of the 10,000 sheep created for aaron Koblin’s thesheepmarket.com by workers for amazon’s mechanical turk who were paid .02
cents to “draw a sheep facing to the left.”
image from thesheePmarket.com By aron koBlin