Vviewpoints
I
M
A
G
E
R
Y
B
Y
L
U
C
K
Y
T
E
A
M
S
T
U
D
I
O
ers were able to successfully deanonymize the papers’ authors. We find that
anonymization is imperfect but fairly
effective: 70%–86% of the reviews were
submitted with no author guesses,
and 74%–90% of reviews were submitted with no correct guesses. Reviewers
who believe themselves to be experts
on a paper’s topic were more likely to
attempt to guess author identities but
no more likely to guess correctly. Overall, we strongly support the continued
use of double-blind review, finding the
extra administrative effort minimal and
well worth the benefits.
PEER REVIEW IS a cornerstone of the academic publication process but can be subject to the flaws of the humans who perform it. Evidence suggests
subconscious biases influence one’s
ability to objectively evaluate work: In
a controlled experiment with two disjoint program committees, the ACM International Conference on Web Search
and Data Mining (WSDM’ 17) found
that reviewers with author information
were 1.76x more likely to recommend
acceptance of papers from famous authors, and 1.67x more likely to recommend acceptance of papers from top
institutions. 6 A study of three years of
the Evolution of Languages conference
(2012, 2014, and 2016) found that, when
reviewers knew author identities, review
scores for papers with male-first authors were 19% higher, and for papers
with female-first authors 4% lower. 4 In a
medical discipline, U.S. reviewers were
more likely to recommend acceptance
of papers from U.S.-based institutions. 2
These biases can affect anyone, regardless of the evaluator’s race and gender. 3 Luckily, double-blind review can
mitigate these effects1, 2, 6 and reduce
the perception of bias, 5 making it a constructive step toward a review system
that objectively evaluates papers based
strictly on the quality of the work.
Three conferences in software engi-
neering and programming languages
held in 2016—the IEEE/ACM Inter-
national Conference on Automated
Software Engineering (ASE), the ACM
International Conference on Object-
Oriented Programming, Systems, Lan-
guages, and Applications (OOPSLA),
and the ACM SIGPLAN Conference on
Programming Language Design and Im-
plementation (PLDI)—collected data on
anonymization effectiveness, which wea
use to assess the degree to which review-
a Sven Apel and Sarfraz Khurshid were the ASE’ 16
PC chairs, Claire Le Goues and Yuriy Brun were
the ASE’ 16 review process chairs, Yannis
Smaragdakis was the OOPSLA’ 16 PC chair,
and Emery Berger was the PLDI’ 16 PC chair.
Viewpoint
Effectiveness of
Anonymization in
Double-Blind Review
Assessing the effectiveness of anonymization in the review process.
DOI: 10.1145/3208157