Figure 4: The find-fix-verify algorithm in Soylent identifies patches in need of
editing, suggests fixes to the patches, and votes on those fixes.
with transcription tasks, it turns out
that showing workers the guesses of
other workers often leads them astray,
especially if the guesses are self-consistent but wrong.
Crowd workers exhibit high variance in the amount of effort they invest
in a task. Some are lazy turkers, who do
as little work as necessary to get paid,
while others are eager beavers, who go
above and beyond the requirements,
either to be helpful or to signal that
they aren’t lazy turkers, but in counterproductive ways. We need new design
patterns for algorithms involving human computation that recognize and
control this behavior.
For example, Soylent uses a find-fix-
verify pattern to improve the quality
of proofreading and document short-
ening (Figure 4). In this pattern, some
workers find problems, other workers
fix them, and still other workers verify
the fixes. But questions remain. What
other algorithms and design patterns
are useful? How should algorithms in-
volving human computation be evalu-
ated and compared from a theoretical
point of view?
Moving from prototyping to actual
deployment requires facing questions
about how to obtain a reliable and well-performing source of human computation for the system. How can we recruit
a crowd to help, and motivate it to continue to help over time, while optimizing for cost, latency, bandwidth, quality, churn, and other parameters?
For paid crowds, these questions in-
tersect with labor economics. Some of
our recent work has found that workers
in human computation markets like
Mechanical Turk behave in unusual
ways. For example, instead of seeking
work that provides a target wage, they
often seek a target earning amount,
and simply work until they reach their
target, consistent with game-playing
behavior [ 5].