Figure 2: In Soylent, after the crowd has suggested words or phrases that can be edited, the end-user can shorten his or her text interactively with a slider. Red text indicates locations where cuts or rewrites have occurred.

Figure 3: Some human computation processes are iterative (left), involving a succession of interleaved improvement steps (by one person) and voting steps (by several people). Other processes are parallel (right), in which individuals generate original content, and voters simply choose among the alternatives.

 

new primitive called “once,” meaning they should only be executed once over all reruns of a program. Subsequent runs of the program check the database before performing operations marked with “once” to see if they have already been executed. This model makes it much easier to code algorithms involving human computation. For example, a TurKit program can sort a list of images using human preference judgments by calling the human computation in the sort algorithm’s comparison function, and wrapping those calls in “once” to make them persistent.

Another programming challenge is the development of algorithms and

design patterns that handle the idio-syncrasies of human beings. Humans are not programmable machines, and they don’t always follow instructions, unintentionally or otherwise. Sometimes this should be embraced and supported, to harness the creativity and diversity of the crowd. Other times, it simply produces noisy, erroneous, or useless results.

For example, we have studied alter-
native algorithms for content creation
[ 4]. Iterative processes are similar to
Wikipedia or open-source software
development. People build on existing
content created by others, with vot-
ing or independent review ensuring
that the process stays on track. Paral-
lel processes are often seen in design
contests, like Threadless.com, where
people generate content independent-
ly, and then the best is chosen through
a vote. See Figure 3.