can ask U to retype a word that an OCR
program has failed to recognize (the
“payment”), thereby contributing to a
CS effort on digitizing written text (that
is, system B). This is the key idea behind
the reCAPTCHA project.
34 The MOBS
project12, 13 employs the same solution.
In particular, it ran experiments where
a user U can access a Web site (such as
a class homepage) only after answering
a relatively simple question (such as, is
string “1960” in “born in 1960” a birth
date?). MOBS leverages the answers to
help build a data integration system.
This solution works best when the “
payment” is unintrusive or cognitively simple, to avoid deterring users from using
The fifth solution is to piggyback on
the user traces of a well-established system (such as building a spelling correction system by exploiting user traces of
a search engine, as discussed previously). This gives us a steady stream of users. But we must still solve the difficult
challenge of determining how the traces can be exploited for our purpose.
Once we have selected a recruitment strategy, we should consider
how to further encourage and retain
users. Many encouragement and retention (E&R) schemes exist. We briefly
discuss the most popular ones. First,
we can provide instant gratification, by
immediately showing a user how his or
her contribution makes a difference.
Second, we can provide an enjoyable experience or a necessary service, such as
game playing (while making a contribution).
32 Third, we can provide ways to
establish, measure, and show fame/trust/
7, 13, 24, 25 Fourth, we can set up
competitions, such as showing top rated users. Finally, we can provide
ownership situations, where a user may feel
he or she “owns” a part of the system,
and thus is compelled to “cultivate”
that part. For example, zillow.com displays houses and estimates their market prices. It provides a way for a house
owner to claim his or her house and
provide the correct data (such as number of bedroomss), which in turn helps
improve the price estimation.
These E&R schemes apply naturally
to volunteering, but can also work well
for other recruitment solutions. For
example, after requiring a set of users
to contribute, we can still provide instant gratification, enjoyable experi-
Given the success
we expect that
this emerging field
will grow rapidly.
ence, fame management, and so on, to
maximize user participation. Finally,
we note that deployed CS systems often
employ a mixture of recruitment methods (such as bootstrapping with “
requirement” or “paying,” then switching to “volunteering” once the system
is sufficiently “mature”).
What contributions can users
make? In many CS systems the kinds
of contributions users can make are
somewhat limited. For example, to
evaluate, users review, rate, or tag; to
share, users add items to a central Web
site; to network, users link to other users; to find a missing boat in satellite
images, users examine those images.
In more complex CS systems, however, users often can make a far wider
range of contributions, from simple
low-hanging fruit to cognitively complex ones. For example, when building a structured KB, users can add a
URL, flag incorrect data, and supply
attribute-value pairs (as low-hanging
3, 5 But they can also supply inference rules, resolve controversial issues, and merge conflicting inputs (as
cognitively complex contributions).
The challenge is to define this range of
possible contributions (and design the
system such that it can gather a critical
crowd of such contributions).
Toward this goal, we should consider four important factors. First,
how cognitively demanding are the
contributions? A CS system often has
a way to classify users into groups,
such as guests, regulars, editors, admins, and “dictators.” We should take
care to design cognitively appropriate
contribution types for different user
groups. Low-ranking users (such as
guests, regulars) often want to make
only “easy” contributions (such as answering a simple question, editing one
to two sentences, flagging an incorrect data piece). If the cognitive load is
high, they may be reluctant to participate. High-ranking users (such as editors, admins) are more willing to make
“hard” contributions (such as resolving controversial issues).
Second, what should be the impact
of a contribution? We can measure the
potential impact by considering how
the contribution potentially affects
the CS system. For example, editing a
sentence in a Wikipedia page largely
affects only that page, whereas revis-