ing an edit policy may potentially affect million of pages. As another example, when building a structured KB, flagging an incorrect data piece typically has less potential impact than supplying an inference rule, which may be used in many parts of the CS system. Quantifying the potential impact of a contribution type in a complex CS system may be difficult. 12, 13 But it is important to do so, because we typically have far fewer high-ranking users such as editors and admins (than regulars, say). To maximize the total contribution of these few users, we should ask them to make potentially high-impact contributions whenever possible.

Third, what about machine contributions If a CS system employs an algorithm for a task, then we want human users to make contributions that are easy for humans, but difficult for machines. For example, examining textual and image descriptions to decide if two products match is relatively easy for humans but very difficult for machines. In short, the CS work should be distributed between human users and machines according to what each of them is best at, in a complementary and synergistic fashion.

Finally, the user interface should make it easy for users to contribute. This is highly non-trivial. For example, how can users easily enter domain knowledge such as “no current living person was born before 1850” (which can be used in a KB to detect, say, incorrect birth dates)? A natural language format (such as in openmind. org) is easy for users, but difficult for machines to understand and use, and a formal language format has the reverse problem. As another example, when building a structured KB, contributing attribute-value pairs is relatively easy (as Wikipedia infoboxes and Freebase demonstrate). But contributing more complex structured data pieces can be quite difficult for naive users, as this often requires them to learn the KB schema, among others. 5

How to combine user contributions?
Many CS systems do not combine con-
tributions, or do so in a loose fashion.
For example, current evaluation sys-
tems do not combine reviews, and com-
bine numeric ratings using relatively
simple formulas. Networking systems
simply link contributions (homepages
and friendships) to form a social net-
work graph. More complex CS systems,
however, such as those that build soft-
ware, KBs, systems, and games, com-
bine contributions more tightly. Exactly
how this happens is application depen-
dent. Wikipedia, for example, lets users
manually merge edits, while ESP does
so automatically, by waiting until two
users agree on a common word.