ranges from relatively simple well-established systems such as reviewing books to complex emerging systems that build structured knowledge bases to systems that “piggyback” onto other popular systems. We discuss fundamental challenges such as how to recruit and evaluate users, and to merge their contributions. Given the space limitation, we do not attempt to be exhaustive. Rather, we sketch only the most important aspects of the global picture, using real-world examples. The goal is to further our collective understanding—both conceptual and practical—of this important emerging topic.

artwork by aaron koblIn and takashI kawashIMa

It is also important to note that many crowdsourcing platforms have been built. Examples include Mechanical Turk, Turkit, Mob4hire, u Test, Freelancer, eLance, oDesk, Guru, Topcoder, Trada, 99design, Inno-centive, CloudCrowd, and Cloud-Flower. Using these platforms, we can quickly build crowdsourcing systems in many domains. In this survey, we consider these systems (that is, applications), not the crowdsourcing platforms themselves.

Crowdsourcing systems

Defining crowdsourcing (CS) systems turns out to be surprisingly tricky. Since many view Wikipedia and Linux as well-known CS examples, as a natural starting point, we can say that a CS system enlists a crowd of users to explicitly collaborate to build a long-lasting artifact that is beneficial to the whole community.

This definition, however, appears too restricted. It excludes, for example, the ESP game, 32 where users implicitly collaborate to label images as a side effect while playing the game. ESP clearly benefits from a crowd of users. More importantly, it faces the same human-centric challenges of Wikipedia and Linux, such as how to recruit and evaluate users, and to combine their contributions. Given this, it seems unsatisfactory to consider only explicit collaborations; we ought to allow implicit ones as well.

The definition also excludes, for example, an Amazon’s Mechanical Turk-based system that enlists users to find a missing boat in thousands of satellite images. 18 Here, users do not build any artifact, arguably nothing is long lasting, and no community exists either

ten thousand Cents is a digital artwork by aaron Koblin that creates a representation of a $100 bill. using a custom drawing tool, thousands of individuals, working in isolation from one another, painted a tiny part of the bill without knowledge of the overall task.

(just users coming together for this particular task). And yet, like ESP, this system clearly benefits from users, and faces similar human-centric challenges. Given this, it ought to be considered a CS system, and the goal of building artifacts ought to be relaxed into the more general goal of solving problems. Indeed, it appears that in principle any non-trivial problem can benefit from crowdsourcing: we can describe the problem on the Web, solicit user inputs, and examine the inputs to develop a solution. This system may not be practical (and better systems may exist), but it can arguably be considered a primitive CS system.

Consequently, we do not restrict the type of collaboration nor the target problem. Rather, we view CS as a general-purpose problem-solving method. We say that a system is a CS system if it enlists a crowd of humans to help solve a problem defined by the system owners, and if in doing so, it addresses the following four fundamental challenges: