Figure 1: With Viz Wiz, blind people take photos using their mobile phones and
submit them alongside a question, spoken orally into the phone, shown here above
each image. A crowd of anonymous users reply, shown below, with response time
given in seconds in parentheses.
ers answer questions they have about
things around them that they cannot
see. The blind person takes a photograph with a smartphone’s camera,
records a spoken question (also using
the phone), and then uploads the query
and picture to a crowd of sighted users
on the net who are better able to ans wer
it (see Figure 1). For example, if a blind
person grabs a can out of her cupboard
but has forgotten what’s inside it, she
can snap a photo of the can and its label, upload it, and ask the sighted users
what’s in the can.
A related system, Sinch [ 7], draws on
the crowd to provide assistance to web-enabled mobile device users who have
situational disabilities, such as the
limited ability to read a small screen,
arthritis or hand tremors that make it
difficult to click on small web page targets, and slow networks. With Sinch,
the mobile users speak a question into
their phone and the crowd searches
the web for answers, using their more
capable desktop web access, and returning web pages with the requested
Another reason to use a crowd is
the “many eyes” principle, which has
been claimed as an advantage of open-source software development (the
complete phrase is “many eyes make
bugs shallow”). We have exploited this
principle in Soylent [ 2], a Microsoft
Word extension that uses a crowd for
proofreading, shortening, and repetitive editing. A typical run of Soylent
may have dozens of people looking at
each paragraph of a document, finding
errors that a single writer might miss.
In fact, a conference paper submitted
about Soylent contained a grammatical error that was overlooked by not
only Word’s built-in grammar checker,
but also eight authors and six reviewers. However, when we passed the paper through Soylent, the crowd caught
A corollary of the many eyes principle is diversity. The fact is, a crowd
comprises a wide range of ideas, opinions, and skills. For example in Soylent,
the system not only indentifies writing
errors, but also suggests multiple ways
to fix them. It can suggest text to cut
to save space—a tough task even for
skilled authors, who are often reluctant to make cuts. Soylent can typi-
What color is this pillow?
(89s) I can’t tell.
(105s) multiple shades
of soft green, blue and
What denomination is
Do you see picnic tables
across the parking lot?
What temperature is my
oven set to?
Can you please tell me
what this can is?
(69s) it looks like 425
degrees but the image
is difficult to see.
(552s) Goya Beans
(99s) no c
cally trim text down to 85 percent of its
original length, without changing the
meaning of the text or introducing errors (see Figure 2).
Prototyping a human computation system is hard if you have to entice a crowd
to visit your website. Games With a Purpose handles this by making the experience fun—but not all human computation systems are fun enough to
be self-motivating, particularly at the
prototyping stage. Mechanical Turk is
a good prototyping platform for many
forms of human computation, because
it offers a ready service for recruiting a
crowd on demand. And the first prototypes for Viz Wiz and Soylent were built
on Mechanical Turk.
Yet thinking about programming
“A group of many
people has abilities
and knowledge that
one single end-user
does not... The fact
is, a crowd comprises
a wide range of
ideas, opinions, and
with human beings inside the system
poses special problems. For example
with Mechanical Turk, a request for
a human to do a small task can take
a few minutes and cost a few cents to
get a result, which is astounding in one
sense (that you can obtain human assistance so quickly and so cheaply), but
is abysmally slow and expensive compared to a conventional function call.
Programmers need new tools that
can help them experiment with human
computation in their systems. For example, our TurKit toolkit [ 3] integrates
Mechanical Turk calls in a traditional
imperative/object-oriented programming paradigm, so that programmers
can write algorithms that incorporate
human computation in a familiar way.
TurKit does this using a novel programming model called “crash and
rerun,” which is suited to long-running
distributed processes where local computation (done by software) is cheap,
and remote work (done by humans) is
The insight of crash-and-rerun programming is that if our program crashes, it is cheap to rerun the entire program up to the place where it crashed.
This is true as long as rerunning does
not re-perform all the costly external
operations from the previous run. The
latter problem is solved by recording
information in a database every time a
costly operation is executed.
Costly operations are marked by a