mechanism goals.
A1
A2
R1
R2
R3
Authors should not submit poor-quality papers
Authors should become reviewers
Reviewers should submit well-substantiated reviews
Reviewers should not favor their friends
Reviewers should not denigrate competing papers
quality of reviews. It also leads to an ever-increasing variance in paper quality. Similarly, as the acceptance rate of a conference declines, there is a greater incentive for reviewers to write overly negative reviews and favor their friends.
some sense of coverage of the field.
Interestingly, the problems outlined here arise because the existing paper reviewing process does not explicitly address these contradictory incentives. There is no explicit incentive for authors to become reviewers or for authors to limit the number of papers they submit, or to submit good-quality papers. There is no check on reviewers who write skimpy reviews,a are overly negative, or play favorites. No wonder the system barely works!
Paper reviewing and publishing can be viewed as a game. There are three players in this game, who are assumed to be rational, in the usual economic sense, and who have the following incentives:
˲ Authors want to get published, or, at least, get detailed, but not necessar-
4 COLOR Process: CMYK PLATE
ily positive, reviewer feedback on their work. They also don’t want to be induced into becoming reviewers.
˲Reviewers/PC members want to minimize their work (for instance, by giving scores, but no justifications), while trying to reject papers that compete with their own papers, and accepting papers from their friends. They want to reject unacceptable papers that would embarrass them. Finally, they want to get the prestige of being in the PC.
˲ Chairs/TCP/Research Community
stakeholders want to have the high-
est quality slate of papers, while trying
a Other than a slight risk of embarrassment at
to include fresh ideas, and providing the PC meeting.
11-08 dtSearch 7 AD CACM: 1/2 page 7 x 4.875 in
mechanisms for incentive alignment Our goals, illustrated in the table here, involve designing mechanisms such that it is incentive-compatible to do the right thing. Here, we describe some mechanisms to achieve these goals (correlated to the A1, A2, R1, R2, R3 labeling scheme established in the table). Our proposals include some steps that have been tried by some brave conference PC chairs. Others that are
Instantly Search Terabytes of Text
◆ Dozens of indexed, unindexed, fielded data and full-text search options (including Unicode support for hundreds of international languages)
◆ File parsers/converters for hit-highlighted display of all popular file types
◆ Spider supports static and dynamic web data; highlights hits while displaying links, formatting and images intact
◆ API supports .NET, C++, Java, databases, etc. New .NET Spider API
Desktop with Spider
Network with Spider
Publish for CD/DVDs
Web with Spider
Engine for Win & .NET
Engine for Linux
The Smart Choice for Text Retrieval® since 1991
◆ “Bottom line: dtSearch manages a terabyte of text in a single index and returns results in less than a second” – Info World
◆ “For combing through large amounts of data,” dtSearch “leads the market” – Network Computing
◆ dtSearch “covers all data sources ... powerful Web-based engines”
– e WEEK
◆ dtSearch “searches at blazing speeds” – Computer Reseller News Test Center
New
64-bit
See www.dtsearch.com for hundreds more reviews, and hundreds of developer case studies
Contact dtSearch for fully-functional evaluations
1-800-IT-FINDS • www.dtsearch.com
References:
Archives