The SIGCSE Symposium: A Brief History
a reviewer outside North America could take a few weeks,
and such mailing delays did not fit within timing constraints
for reviewing. Consequently, before SIGCSE 2000, papers
could not be reviewed by computing educators outside
North America. With the possibility of online submissions for
SIGCSE 2000, however, electronic submissions could be assigned to reviewers world-wide—greatly extending the range
of people who could participate in the reviewing process.
• Reviewer Variability Study: When many reviewers evaluate
papers, questions naturally arise regarding variability in
ratings from one reviewer to the next. To help address this
question, as part of system development for SIGCSE 2000,
10 papers (largely selected at random) were assigned to
100 reviewers each, in addition to the normal process of
reviewing all the submissions. In addition to basic questions
regarding overall rating variability, the study also considered
what impact several variables might have on overall ratings.
In summary, the findings are largely reassuring, but also
may suggest directions for further exploration.
In summary, “the following factors are NOT statistically
significant in contributing to overall paper ratings:
referee gender, referee’s country (U.S. versus non-U.S.),
familiarity of referee with subject, or paper format
(electronic submission versus hardcopy).” [ 13, p. 181]
Papers with either low or high median scores had
relatively low standard deviation scores—reviewers often
agreed that papers were quite good or quite weak.
Papers with intermediate median scores displayed higher
standard deviation scores; reviewers had relatively
broader reactions to papers that were not obviously
wonderful or obviously weak.
Reference [ 13] provides a more complete discussion of this
• Timing of Author (and Reviewer) Submissions: Before
SIGCSE 2000, all paper submissions had to be mailed.
Thus, to be confident that mail would arrive in time for the
reviewing process, authors typically completed and mailed
their manuscripts several days before a deadline. Even if
consideration of a submission was based upon a postmark,
the paper still had to arrive within a time interval, so it could
be distributed and reviewed. Pushing a deadline too closely
might result in a paper not being eligible for consideration.
Electronic submissions, naturally, changed such timing
considerations [ 10, 11]. Papers could be submitted the night
of a deadline and still be included within the reviewing pro-
cess. As a result, many authors would submit papers close
to a stated deadline. By SIGCSE 2010 and SIGCSE 2011, the
clear majority of papers were submitted within a few days,
or even within a few hours, of the stated deadline. For exam-
ple, the graph in Figure 5 shows the number of submissions
received in the days before the deadline for these two con-
ferences. In this graph, the horizontal axis measures the days
Within just a few years, each paper could be sent to six
or seven reviewers to obtain a particularly broad range of
• SIGCSE 2009–present: Although an online system
allowed extensive feedback for papers by many reviewers,
nervousness was sometimes expressed regarding divergent
views and ratings. Inherently within the process, reviewers
provided their views about each paper, but reviewers did
not have an opportunity to compare differing perspectives,
to discuss strengths and weaknesses, and to work toward
a consensus. In response, the reviewing process was
expanded to yield a multi-tiered system.
Initially with SIGCSE 2009—a small group of “Associate
Program Chairs” served to oversee sets of papers—
reading reviewers’ reviews and providing a second stage
in the overall reviewing process. By handling several
papers, this second tier could add a global perspective to
the individual papers and reviews.
Shortly thereafter, a group of people were designated as
“meta-reviewers,” each of whom read both a group of
papers and their reviews, and then prepared an informed
summary and reflection for each paper. Overall, a
“meta-review” could highlight perspectives and insights
from several reviewers, while also providing a broad
Within a few years the meta-reviewing process was
expanded to include discussion of each paper by all its
reviewers, under the direction of an “Associate Program
Chair” (APC). With discussion, reviewers might refine
their own reviews, adjust their ratings, and/or provide
insights. Often, this process reduced variability and
provided authors (and the program chairs) with clearer
direction and feedback [ 3].
5. 3. FURTHER CONSEQUENCES FROM ONLINE
As already noted, the shift to the online submission and reviewing of papers for SIGCSE 2000 and later resolved many logistical challenges and yielded several anticipated benefits. Three
additional elements may or may not have been anticipated.
• International Authors and Reviewers: The paper-based
system in place before SIGCSE 2000 relied upon postal mail
for the transmission of papers, both for submission and
for reviewing. Within North American, postal mail took
some time, but mail usually was delivered within a few days.
However, mail to and from Europe or other international
destinations often was much slower. As a result, authors
from outside North America often had to mail their
submissions significantly before a stated deadline.
Further, mail from a program chair (in North America) to