ranges from relatively simple well-established systems such as reviewing
books to complex emerging systems
that build structured knowledge bases to systems that “piggyback” onto
other popular systems. We discuss
fundamental challenges such as how
to recruit and evaluate users, and to
merge their contributions. Given the
space limitation, we do not attempt
to be exhaustive. Rather, we sketch
only the most important aspects of
the global picture, using real-world
examples. The goal is to further our
collective understanding—both conceptual and practical—of this important emerging topic.
artwork by aaron koblIn and takashI kawashIMa
It is also important to note that
many crowdsourcing platforms have
been built. Examples include Mechanical Turk, Turkit, Mob4hire, u Test, Freelancer, eLance, oDesk, Guru,
Topcoder, Trada, 99design, Inno-centive, CloudCrowd, and Cloud-Flower. Using these platforms, we
can quickly build crowdsourcing
systems in many domains. In this
survey, we consider these systems
(that is, applications), not the
crowdsourcing platforms themselves.
Defining crowdsourcing (CS) systems
turns out to be surprisingly tricky.
Since many view Wikipedia and Linux
as well-known CS examples, as a natural starting point, we can say that a
CS system enlists a crowd of users to
explicitly collaborate to build a long-lasting artifact that is beneficial to the
This definition, however, appears
too restricted. It excludes, for example,
the ESP game,
32 where users implicitly
collaborate to label images as a side
effect while playing the game. ESP
clearly benefits from a crowd of users.
More importantly, it faces the same human-centric challenges of Wikipedia
and Linux, such as how to recruit and
evaluate users, and to combine their
contributions. Given this, it seems unsatisfactory to consider only explicit
collaborations; we ought to allow implicit ones as well.
The definition also excludes, for example, an Amazon’s Mechanical Turk-based system that enlists users to find
a missing boat in thousands of satellite
18 Here, users do not build any
artifact, arguably nothing is long lasting, and no community exists either
ten thousand Cents is a digital artwork by
aaron Koblin that creates a representation
of a $100 bill. using a custom drawing tool,
thousands of individuals, working in isolation
from one another, painted a tiny part of the
bill without knowledge of the overall task.
(just users coming together for this
particular task). And yet, like ESP, this
system clearly benefits from users, and
faces similar human-centric challenges. Given this, it ought to be considered
a CS system, and the goal of building
artifacts ought to be relaxed into the
more general goal of solving problems.
Indeed, it appears that in principle any
non-trivial problem can benefit from
crowdsourcing: we can describe the
problem on the Web, solicit user inputs, and examine the inputs to develop a solution. This system may not be
practical (and better systems may exist), but it can arguably be considered
a primitive CS system.
Consequently, we do not restrict
the type of collaboration nor the target
problem. Rather, we view CS as a general-purpose problem-solving method.
We say that a system is a CS system if
it enlists a crowd of humans to help solve
a problem defined by the system owners,
and if in doing so, it addresses the following four fundamental challenges: