ences strongly support the continued
use of double-blind review, find it effective at mitigating (both conscious
and subconscious) bias in reviewing,
and judge the extra administrative
burden to be relatively minor and well
worth the benefits. Technological advances and the now-developed author
instructions reduce the burden. Having a dedicated organizational position to support double-blind review
can also help. The ASE and OOPSLA PC
chairs point out some benefits of revealing author identities midprocess,
while the PLDI PC chair argues some of
those benefits can be preserved in a full
double-blind review process that only
reveals the author identities of accepted papers, while providing significant
additional benefits, such as mitigating
bias throughout the entire process and
preserving author anonymity for rejected paper resubmissions.
1. Budden, et al. Double-blind review favours increased
representation of female authors. Trends in Ecology
and Evolution 23, 1 (Jan. 2008), 4–6.
2. Gastroenterology, Bethesda, MD, USA. U.S. and non-U.S. submissions: An analysis of reviewer bias. JAMA
280, 3 (July 1998), 246–247.
3. Moss-Racusin, C.A. et al. Science faculty’s subtle
gender biases favor male students. PNAS 109, 41
(Apr. 2014), 16474–16479.
4. Roberts, S.G. and Verhoef, T. Double-blind reviewing
at EvoLang 11 reveals gender bias. J. of Language
Evolution 1, 2 (Feb. 2016), 163–167.
5. Snodgrass, R. Single-versus double-blind reviewing:
An analysis of the literature. SIGMOD Record 35, 3
(May 2006), 8–21.
6. Tomkins, A., Zhang, M., and Heavlin, W. D. Single
versus double-blind reviewing at WSDM 2017. CoRR,
Claire Le Goues ( firstname.lastname@example.org) is an Assistant
Professor in the School of Computer Science at Carnegie
Mellon University, Pittsburgh, PA, USA.
Yuriy Brun ( email@example.com) is an Associate
Professor in the College of Information and Computer
Sciences at the University of Massachusetts Amherst,
Amherst, MA, USA.
Sven Apel ( firstname.lastname@example.org) is a Professor of
Computer Science and Chair of Software Engineering in
the Department of Informatics and Mathematics at the
University of Passau.
Emery Berger ( email@example.com) is a Professor in
the College of Information and Computer Sciences at the
University of Massachusetts Amherst, Amherst, MA, USA.
Sarfraz Khurshid ( firstname.lastname@example.org) is a
Professor in Electrical and Computer Engineering at the
University of Texas at Austin, Austin, TX, USA.
Yannis Smaragdakis ( email@example.com) is
a Professor in the Department of Informatics at
the University of Athens.
Kathryn McKinley suggested an early draft of the reviewer
questions used by OOPSLA and PLDI. This work is
supported in part by the National Science Foundation
under grants CCF-1319688, CCF-1453474, CCF-1563797,
CCF-1564162, and CNS-1239498, and by the German
Research Foundation (AP 206/6).
Copyright held by authors.
PC Chairs’ Observations
After completing the process, the PC
chairs of all three conferences reflected on the successes and challenges of
double-blind review. All PC chairs were
strongly supportive of continuing to use
double-blind review in the future. All felt
that double-blind review mitigated effects of (subconscious) bias, which is the
primary goal of using double-blind review. Some PC members also felt so, indicating anecdotally that they were more
confident their reviews and decisions
had less bias. One PC member remarked
that double-blind review is liberating,
since it allows for evaluation without
concern about the impact on the careers
of people they know personally.
All PC chairs have arguments in
support of their respective decisions
on the timing of revealing the authors
(that is, after review submission, before PC meeting, or only for accepted
papers). The PLDI PC chair advocated
strongly for full double-blind, which
enables rejected papers to be anonymously resubmitted to other double-blind venues with common reviewers,
addressing one cause of deanonymization. The ASE PC chairs observed that
in a couple of cases, revealing author
identities helped to better understand
a paper’s contribution and value. The
PLDI PC chair revealed author identities on request, when deemed absolutely necessary to assess the paper.
This happened extremely rarely, and
could provide the benefit observed
by the ASE PC chairs without sacrificing other benefits. That said, one PC
member remarked that one benefit
of serving on a PC is learning who is
working on what; full anonymization
eliminates learning the who, though
still allows learning the what.
Overall, none of the PC chairs felt the
extra administrative burden imposed by
double-blind review was large. The ASE
PC chairs recruited two review process
chairs to assist, and all felt the effort re-
quired was reasonable. The OOPSLA PC
chair noted the level of effort required
to implement double-blind review, in-
cluding the management of conflicts of
interest, was not high. He observed that
it was critical to provide clear guidance
to the authors on how to anonymize pa-
pers (for example, http://2016.splash-
allowed authors to either anonymize
artifacts (such as source code) or to sub-
mit non-anonymized versions to the PC
chair, who distributed to reviewers when
appropriate, on demand. The PC chair
reported this presented only a trivial ad-
ditional administrative burden.
The primary source of additional
administration in double-blind review
is conflict of interest management.
This task is simplified by conference
management software that straightforwardly allows authors and reviewers
to declare conflicts based on names
and affiliations, and chairs to quickly
cross-check declared conflicts. ASE PC
chairs worked with the CyberChairPro
maintainer to support this task. Neither ASE nor OOPSLA observed unanticipated conflicts discovered when
author identities were revealed. The
PLDI PC chair managed conflicts of interest more creatively, creating a script
that validated author-declared conflicts by emailing PC members lists of
potentially conflicted authors mixed
with a random selection of other authors, and asking the PC member to
identify conflicts. The PC chair examined asymmetrically declared conflicts
and contacted authors regarding their
reasoning. This identified erroneous
conflicts in rare instances. None of the
PC chairs found identifying conflicts
overly burdensome. The PLDI PC chair
reiterated that the burden of full double-blind reviewing is well worth maintaining the process integrity throughout the entire process, and for future
Data from ASE 2016, OOPSLA 2016, and
PLDI 2016 suggest that, while anonymization is imperfect, it is fairly effective. The PC chairs of all three confer-
All PC chairs were
of continuing to use
in the future.