July, or September of 2011. The April and July review processes yield accepts, rejects, and revise and resubmits. Submissions in September are either accepted or rejected. The Asian Conference on Machine Learning (ACML2012) had two deadlines and the same model.

The ACM Computer Supported Cooperative Work 2012 (CSCW 2012) conference employed a single submission date and a five-week revision period. This is essentially a rapid version of a journal special issue process: a submission deadline followed by reviewing and one round of revision that is fully examined by the original reviewers. This process is discussed below. It has now been used by ACM Interactive Tabletops and Surfaces 2012, SIGMOD 2013, and CSCW 2013.

Adding a revision cycle resembles a time-compressed variant of the usual practice, where a rejected paper can be resubmitted in a year, but the dynamic is different. A reviewer does not have to make an often stressful, binary, in-or-out recommendation. Reviewers have more incentive to provide constructive, complete reviews, rather than primarily identify flaws that justify rejection. Authors who will get the same reviewers have more incentive to respond to suggestions than when they resubmit a rejected paper to a different conference with a new set of reviewers. Reviewers have to examine some submissions twice, but most find the second review easier and are rewarded by seeing authors take their suggestions seriously. When a rejected paper is resubmitted elsewhere, the overall reviewing burden for the community is greater and less rewarding.

The CSCW 2012 experiment. As program chairs for this large annual conference, we were asked to move the submission deadline forward two months to reduce the reviewing overlap with the larger CHI conference. To turn this lemon into lemonade, we inserted the revision cycle.

Computer Supported Cooperative Work conferences averaged 256 submissions and 56 acceptances (22%) from 2000–2011. In recent years, most submissions were reviewed by two program committee members (called Associate Chairs or ACs) and three external reviewers, followed by a face-to-face meeting of the ACs.

table 1. approaches that merge conference and journal elements.

journal acceptance precedes conference presentation.

Shepherded conference papers become journal articles.

Conference reviewing incorporates a revision cycle.

table 2. adding a revision cycle.

Benefits

Higher quality.

Higher quantity and attendance.

Zero-sum game eliminated.

More constructive review focus. Reviewers see results of effort.

Few decisions passed to committee.

Challenges

More periods of review activity.

Perception of acceptance rate.

More conference tracks?

Reviewer assignment still hard.

 

The 2012 conference received a record 415 submissions. In the first round, each submission was reviewed by an AC and two external reviewers. Forty-five percent were rejected and authors notified. The authors of the remaining 55% had five weeks to upload a revision and a document describing their responses to reviewer comments. A few withdrew; the others revised, often extensively. The revisions were reviewed by the original reviewers. After online discussion only 26 remained unresolved. The face-to-face program committee meeting may have been unnecessary, but it had been scheduled. Final decisions were made there.

In the second round, 27% of the revisions were rejected. Overall, 39.5% of the original submissions were judged to have cleared the quality bar. The traditional process would have accepted approximately 22% ‘as is’; this year all papers were revised. The consensus was that a high-quality conference had become larger and much stronger.

Analysis of the CSCW 2011 data enabled us to focus reviewing where it was needed—we did not increase the overall reviewing burden. In 2011, 60% of submissions found no advocate in the early reviews and not one of those was ultimately accepted. Yet each received a summary review by one AC, was read by a second AC, and was given a rebuttal opportunity. For CSCW 2012, 45% had no advocate in the first round and were summarily but politely rejected. The average workload per submission was reduced despite each “revise and resubmit” paper receiving eight reviews by four reviewers over the two rounds.

The number of reviews or serious considerations fell from an average of 5. 5 per submission the previous year to 4. 6, a reduction of 400. And 35% of all reviews were of revisions, aided by the authors’ documents describing the changes. The rebuttal process was not needed; dropping it led to no objections. Finally, the 60–80 papers that were accepted because of the revision process would have been prime candidates for resubmission elsewhere, so the community was spared hundreds of additional reviews downstream. (Many conferences already have more streamlined review processes than we started with, but could still realize a net reduction in effort by adopting a revision cycle.)

Positive outcomes. First the good news. Overall the reports were very positive. The revise-and-resubmit option for borderline papers in the first round reduced reviewer stress. Reviewers could focus on finding what might be of interest and formulating constructive guidance for revision, rather than identifying flaws that warrant rejection. Reviewers found it rewarding to see authors who had responded well to comments. An interesting benefit was that acceptance was not a zero-sum game. We did not have a quota, just the goal of keeping the same quality bar. Without the pressure to reject 75% to 80% of submissions, a four-per-son review team could iterate with an author without disadvantaging other authors. Some review teams engaged in protracted online discussions. With few decisions left to make, half of the face-to-face meeting was spent discussing broader issues and planning