Vviewpoints

I M A G E R Y B Y L U C K Y T E A M S T U D I O

ers were able to successfully deanonymize the papers’ authors. We find that anonymization is imperfect but fairly effective: 70%–86% of the reviews were submitted with no author guesses, and 74%–90% of reviews were submitted with no correct guesses. Reviewers who believe themselves to be experts on a paper’s topic were more likely to attempt to guess author identities but no more likely to guess correctly. Overall, we strongly support the continued use of double-blind review, finding the extra administrative effort minimal and well worth the benefits.

PEER REVIEW IS a cornerstone of the academic publication process but can be subject to the flaws of the humans who perform it. Evidence suggests subconscious biases influence one’s ability to objectively evaluate work: In a controlled experiment with two disjoint program committees, the ACM International Conference on Web Search and Data Mining (WSDM’ 17) found that reviewers with author information were 1.76x more likely to recommend acceptance of papers from famous authors, and 1.67x more likely to recommend acceptance of papers from top institutions. 6 A study of three years of the Evolution of Languages conference (2012, 2014, and 2016) found that, when reviewers knew author identities, review scores for papers with male-first authors were 19% higher, and for papers with female-first authors 4% lower. 4 In a medical discipline, U.S. reviewers were more likely to recommend acceptance of papers from U.S.-based institutions. 2

These biases can affect anyone, regardless of the evaluator’s race and gender. 3 Luckily, double-blind review can mitigate these effects1, 2, 6 and reduce the perception of bias, 5 making it a constructive step toward a review system that objectively evaluates papers based strictly on the quality of the work.

Three conferences in software engi-
neering and programming languages
held in 2016—the IEEE/ACM Inter-
national Conference on Automated
Software Engineering (ASE), the ACM
International Conference on Object-
Oriented Programming, Systems, Lan-
guages, and Applications (OOPSLA),
and the ACM SIGPLAN Conference on
Programming Language Design and Im-
plementation (PLDI)—collected data on
anonymization effectiveness, which wea
use to assess the degree to which review-
a Sven Apel and Sarfraz Khurshid were the ASE’ 16
PC chairs, Claire Le Goues and Yuriy Brun were
the ASE’ 16 review process chairs, Yannis
Smaragdakis was the OOPSLA’ 16 PC chair,
and Emery Berger was the PLDI’ 16 PC chair.

Viewpoint
Effectiveness of
Anonymization in
Double-Blind Review
Assessing the effectiveness of anonymization in the review process.

DOI: 10.1145/3208157