mechanism goals.

A1

A2

R1

R2

R3

Authors should not submit poor-quality papers

Authors should become reviewers

Reviewers should submit well-substantiated reviews

Reviewers should not favor their friends

Reviewers should not denigrate competing papers

 

quality of reviews. It also leads to an ever-increasing variance in paper quality. Similarly, as the acceptance rate of a conference declines, there is a greater incentive for reviewers to write overly negative reviews and favor their friends.

some sense of coverage of the field.

Interestingly, the problems outlined here arise because the existing paper reviewing process does not explicitly address these contradictory incentives. There is no explicit incentive for authors to become reviewers or for authors to limit the number of papers they submit, or to submit good-quality papers. There is no check on reviewers who write skimpy reviews,^a are overly negative, or play favorites. No wonder the system barely works!

the Paper Publishing Game

Paper reviewing and publishing can be viewed as a game. There are three players in this game, who are assumed to be rational, in the usual economic sense, and who have the following incentives:

˲ Authors want to get published, or, at least, get detailed, but not necessar-

4 COLOR Process: CMYK PLATE

ily positive, reviewer feedback on their work. They also don’t want to be induced into becoming reviewers.

˲Reviewers/PC members want to minimize their work (for instance, by giving scores, but no justifications), while trying to reject papers that compete with their own papers, and accepting papers from their friends. They want to reject unacceptable papers that would embarrass them. Finally, they want to get the prestige of being in the PC.

˲ Chairs/TCP/Research Community
stakeholders want to have the high-
est quality slate of papers, while trying
a Other than a slight risk of embarrassment at
to include fresh ideas, and providing the PC meeting.

11-08 dtSearch 7 AD CACM: 1/2 page 7 x 4.875 in

mechanisms for incentive alignment Our goals, illustrated in the table here, involve designing mechanisms such that it is incentive-compatible to do the right thing. Here, we describe some mechanisms to achieve these goals (correlated to the A1, A2, R1, R2, R3 labeling scheme established in the table). Our proposals include some steps that have been tried by some brave conference PC chairs. Others that are

 

Instantly Search Terabytes of Text

◆ Dozens of indexed, unindexed, fielded data and full-text search options (including Unicode support for hundreds of international languages)

◆ File parsers/converters for hit-highlighted display of all popular file types

◆ Spider supports static and dynamic web data; highlights hits while displaying links, formatting and images intact

◆ API supports .NET, C++, Java, databases, etc. New .NET Spider API

Desktop with Spider

Network with Spider

Publish for CD/DVDs

Web with Spider

Engine for Win & .NET

Engine for Linux

The Smart Choice for Text Retrieval^® since 1991

◆ “Bottom line: dtSearch manages a terabyte of text in a single index and returns results in less than a second” – Info World

◆ “For combing through large amounts of data,” dtSearch “leads the market” – Network Computing

◆ dtSearch “covers all data sources ... powerful Web-based engines”

– e WEEK

◆ dtSearch “searches at blazing speeds” – Computer Reseller News Test Center

New
64-bit

See www.dtsearch.com for hundreds more reviews, and hundreds of developer case studies

Contact dtSearch for fully-functional evaluations

1-800-IT-FINDS • www.dtsearch.com