Communications of the ACM

“suddenly, when
you need a whole
bunch of people to
do something useful,
platforms like ours
let you organize
them very well.”

Still, not everyone in the crowdsourcing space believes abuses are a major concern.

At CrowdFlower, founder and CEO Lukas Biewald observes that crowdsourcing gives people the opportunity to choose those tasks they want to perform when they want to perform them, and earn money or coupons or in-game currency for completing those tasks.

“That seems like a good deal to
me,” he says. “We have over four mil-
lion people working on CrowdFlower
syndicated tasks, and I log into our
partner sites all the time to make sure
that it’s a good experience for every-
one. There are bound to be some com-
plaints and some issues, but we work
as hard as we can to resolve the issues
as quickly as possible.”
CrowdFlower’s business model is
different from that of MobileWorks;
instead of posting jobs on its own site,
it uses an API to syndicate tasks from
customers and then farm them out to
thousands of partner sites, one of them
being MTurk.

Like Turkopticon and MobileWorks, CrowdFlower uses technology—but mainly to guarantee the quality of the work for the employer.

Biewald explains that having a background in AI enabled him write the statistical algorithms that predict the likelihood that a task will be done correctly—and how many checks and redundancies need to be built into the tasks to produce optimal results.

“We have the work history of the
people who sign on,” he says, “so many
of our decisions are based on their his-
torical performance. For instance, if we
think someone is going to be 90% ac-
curate and the customer requires 99%
accuracy, we know we need to have a
second person check the work.”
Also, he says, there are other indica-
tors—if a person works too quickly or
has never done a similar task before,
the software sends up a flag.

Further Reading

von Ahn, L., “human Computation,” (video), August

22, 2012, http://www.youtube.com/ watch?v=tx082gDwGcM

Cushing, E., “Dawn Of The Digital Sweatshop,” August

1, 2012, “East Bay Express,” http://www. eastbayexpress.com/oakland/dawn-of-the- digital-sweatshop/Content?oid=3301022

Kittur, A., Nickerson, J., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., and Horton, J., “The Future of Crowd Work,” published February 2013 at the 16th ACM Conference on Computer Supported Cooperative Work, http://hci.stanford.edu/publications/2013/ CrowdWork/futureofcrowdwork-cscw2013.pdf

Neumann, E., “Tech Company CrowdFlower Denies Labor Violation,” December 4, 2012, “MissionLocal,” http://missionlocal. org/2012/12/tech-company-crowdflower- denies-underpaying-workers/ Ipeirotis, P., “Mechanical Turk: now with 40.92% spam,” December 16, 2010, http://www.behind-the- enemy-lines.com/2010/12/mechanical-turk- now-with-4092-spam.html

Irani, L., and Silverman, M., “Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk,” April 2013, ACM SIGChI Conference on human Factors in Computing Systems, http://www.ics.uci.edu/~lirani/Irani-Silberman-Turkopticon-camready.pdf

Van Pelt, C., and Sorkin, A., “Designing a Scalable Crowdsourcing Platform,” 2012 ACM SIGMOD International Conference on Management of Data, http:// dl.acm.org/citation.cfm?id=2213951

Biewald, L., “Massive Multiplayer human Computation for Fun, Money, and Survival,” ICWE 2011, http://link.springer.com/ chapter/10.1007%2F978-3-642-27997- 3_ 18?LI=true# Hester, V., Shaw, A,. and Biewald, L., “Scalable crisis relief: Crowdsourced SMS translation and categorization with Mission 4636,” ACM Symposium on Computing for Development 2010, http://dl.acm.org/ citation.cfm?id=1926199