XRDS • FALL 2017 • VOL. 24 • NO. 1
quality threshold a worker must meet to
receive the bonus.
Since addressing low-quality work
is one of the biggest open challenges
in crowdsourcing, this line of research
opens up the opportunity to put
crowdsourcing to use to solve the
big problems that cannot be solved by
Parts of this article were built on excerpts from a longer survey paper on
crowdsourcing [ 8]. Special thanks to
my collaborators on the performance-based payment studies: Chien-Ju Ho,
Alex Slivkins, and Sid Suri.
[ 1] Ho, C.-J., Slivkins, A., Suri, S., and Vaughan, J. W.
Incentivizing high quality crowd work. In Proceedings
of the 24th International Conference on World Wide
Web ( W W W ‘ 15). International World Wide Web
Conferences Steering Committee, S witzerland.
[ 2] Harris, C. G. You’re hired! An examination of
crowdsourcing incentive models in human
resource tasks. In Proceedings of the Workshop on
Crowdsourcing for Search and Data Mining (CSDM)
at the Fourth ACM International Conference on Web
Search and Data Mining ( WSDM) (Hong Kong, Feb.
9). ACM, New York, 2011.
[ 3] Yin, M., Chen, Y., and Sun, Y. Monetary interventions
in crowdsourcing task switching. In Second
AAAI Conference on Human Computation and
[ 4] Sha w, A. D., Horton, J. J. and. Chen, D. L. Designing
incentives for inexpert human raters. In Proceedings
of the ACM 2011 Conference on Computer Supported
Cooperative Work (CSC W ‘ 11). ACM, Ne w York, 2011,
[ 5] Yin, M., Chen, Y., and Sun, Y. The effects of
performance-contingent financial incentives in
online labor markets. In Proceedings of the 27th AAAI
Conference on Artificial Intelligence (AAAI’ 13). AAAI
Press, 2013, 1191-1197.
[ 6] Chandler, D. and Kapelner, A. Breaking monotony
with meaning: Motivation in crowdsourcing markets.
Journal of Economic Behavior and Organization 90,
[ 7] Law, E., Yin, M., Goh, J., Chen, K., Terry, M., and Gajos.
K. Z. Curiosity killed the cat, but makes crowd work
better. In Proceedings of the 2016 CHI Conference
on Human Factors in Computing Systems (CHI ‘ 16).
ACM, New York, 2016, 4098-4110
[ 8] Vaughan, J. W. Making better use of the crowd.
Working paper. 2017; http: // www.jennwv.com/
Dr. Jennifer Wortman Vaughan is a senior researcher at
Microsoft Research, New York City. She is the recipient
of a National Science Foundation CAREER award, a
Presidential Early Career Award for Scientists and
Engineers, and a handful of best paper or best student
paper awards. She also co-founded the Annual Workshop
for Women in Machine Learning, which has been held
each year since 2006 and is expected to attract close to a
thousand registered participants this year.
© 2017 Copyright held by owners/authors.
Publication rights licensed to ACM
directly measurable. To quantify which
tasks are effort-responsive, we used
time as a proxy for effort. In particular,
we looked at the relationship between
the time it took each worker to complete a task and that worker’s quality
to see whether quality improved with
time. We found tasks like proofreading and spotting differences in images
are effort-responsive, while handwriting recognition and audio transcription are not. Our experiments revealed performance-based payments
led to improved quality on the effort-responsive tasks, but not on others.
Further work is needed before we can
say anything for certain, but these experiments do suggest whether a task
is effort-responsive may play a role in
whether quality can be improved using
I’ve explained how carefully designed
monetary incentives like bonus payments can be used to improve the quality of crowdwork. I’ll finish up by mentioning a few pieces of recent research
that study non-monetary incentives and
intrinsic motivation in crowdsourcing.
As a first example, recent research
has suggested crowdworkers are more
active when the tasks they’re asked to
complete are framed as meaningful [ 8].
This might partially explain the popu-
larity and success of citizen-science
platforms like the Zooniverse or Science
at Home. In one study, Mechanical Turk
workers were recruited to label medical
images. Each worker was randomly as-
signed to one of three treatments. In
the first, workers were given no context
about the task. In the second, workers
were told they were labeling tumor cells
and the results of their work would be
used to assist medical researchers. In
the final treatment, workers were again
given no context for the task, and were
additionally told the labels they gener-
ated would not be recorded; that is, all
of their work would immediately be dis-
carded. The authors found when work-
ers were told their work would benefit
medical research, the quantity of work
they produced increased, but, perhaps
surprisingly, their work was not sig-
nificantly more accurate. On the other
hand, when workers were told their
work would be discarded, the quality of
their work was significantly lower than
other treatments, though the quantity
was not affected.
As a second example, another recent study looked at how curiosity can
be used as a driving factor to motivate
crowdworkers [ 7]. This study was inspired by the information gap theory of
curiosity, which suggests when people
are made aware there is a gap in their
knowledge, they actively seek out the
information needed to fill in this gap.
The authors proposed several “
curiosity interventions” aimed at stoking
workers’ curiosity. While some interventions increased worker productivity,
there is some subtlety in how to most
effectively engage workers’ curiosity.
It is an interesting open direction to
determine how to best design tasks to
keep workers engaged while encouraging high-quality output.
We’ve explored how both monetary and
non-monetary incentives can be used
to combat the problem of low-quality
crowdwork. In particular, we dove into
the details of a recent study aimed at
examining when, why, and on which
tasks performance-based payments
lead to quality improvements on one
particular crowdsourcing platform,
Amazon Mechanical Turk. We saw performance-based payments can improve
quality for effort-responsive tasks. To
be effective, bonus payments should
be large relative to the base payment.
Luckily, as long as the bonus payment
is large enough to make the reward
salient, the precise amount doesn’t
matter too much, nor does the precise
If workers think their
work is likely to be
rejected if it is not
high enough quality,
then they might act
as if payments are
even if they’re not.