Communications - March 2009

Society | DOI: 10.1145/1467247.1467254
Leah Hoffmann
crowd control
Using crowdsourcing applications, humans around the world are transcribing
audio files, conducting market research, and labeling data, for work or pleasure.

ThoUGh CoMpUteRS have outstripped us in arithmetic and chess, there are still plenty of areas where the human mind excels, such as visual cognition and language processing. And if one mind is good, as the proverb goes, two—or two thousand—are often better. That insight, and its consequences, drew worldwide interest with the 2004 publication of James Surowiecki’s best-selling The Wisdom of Crowds, which argued that a large group of people are superior at certain types of rational tasks than individuals or even experts.

Now researchers are turning to computers to help us take advantage of our own cognitive abilities and of the wisdom of crowds. Through a distributed problem-solving process variously known as crowdsourcing, human computation, and computer-aided micro-consulting, answers are solicited online to a set of simple, specific questions that computers can’t solve. Is this a picture of a fish? Do you like that style of shoe? How many hotels are on St. George’s Island, and which ones have Internet access?

The amateur, often anonymous workers who agree to execute these tasks are usually given some sort of social or financial incentive. A few cents might buy the answer to a simple data-labeling task, while a more ardu-

ous job like audio transcription could require a couple of dollars. Reposition the task as a game, and many people even “work” for free. Either way, the possibilities—for creating corpuses of annotated data, conducting market research, and more—have both computer scientists and companies excited.

One of the oldest commercial crowdsourcing applications is Amazon’s Mechanical Turk. Named after a famous 18 century chess-playing th

“machine” that was secretly operated by a human, it offers a flexible, Web-based platform for creating and publicizing tasks and distributing micro-payments. Since its launch in 2006, Turk has spawned both a vocabulary and a mini-marketplace. Workers, or “Turkers” (there are more than 200,000 in 185,000 countries, according to Amazon), select “Human Intelligence Tasks” (HITs) that match their interests and abilities. Motivations vary. Some work odd hours or at night to generate extra income, while others simply desire a more productive way to kill time online, like solitaire with financial rewards. As in the of-fline world, more money buys faster results, and Amazon’s HIT requesters often experiment to find a pay scale that matches their needs.

Also part of the Turk economy are companies like Dolores Labs and Cast-ingWords, which rely on Amazon’s

technology to power their own crowdsourcing applications. Dolores Labs, based in San Francisco, posts Turk HITs on behalf of its clients, then filters the answers through custom-built software systems to check for quality and generate meaningful results. Data is ultimately used to perform tasks like filter comment spam, tag data for search engine optimization, and research market trends.

“Many companies don’t have the resources to describe tasks, put them up online, and manage the data they get,” explains Lukas Biewald, the company’s founder and CEO. Nor do they have time for Dolores’s extensive quality-control measures, which include creating “test” questions whose answers are already known, checking responses against one another, tracking individual answer histories, and creating a confidence measure with which to weight the resulting data.

Dolores also guides clients through the many variables that are involved in designing a crowdsourced project. How arduous is each task? How quickly are results needed? How would clients like to deal with the statistical outliers that are caught by Dolores’ quality-control algorithms? If you’re checking user-generated content for pornography, for example, you might err on the side of caution.

According to Biewald’s estimates,

some of the 10,000 sheep created for aaron Koblin’s thesheepmarket.com by workers for amazon’s mechanical turk who were paid .02 cents to “draw a sheep facing to the left.”