pers had no reviewers correctly guess
even one author, and most reviews
contained no correct guess (ASE 90%,
OOPSLA 74%, PLDI 81%).
Are experts more likely to guess and
guess correctly? All reviews included a
self-reported assessment of reviewer
expertise (X for expert, Y for knowledgeable, and Z for informed outsider).
Figure 2 summarizes guess incidence
and guess correctness by reviewer expertise. For each conference, X reviewers were statistically significantly more
likely to guess than Y and Z reviewers
(p ≤ 0.05). But the differences in guess
correctness were not significant, ex-
Methodology
The authors submitting to ASE 2016,
OOPSLA 2016, and PLDI 2016 were instructed to omit author information
from the author block and obscure, to
the best of their ability, identifying information in the paper. PLDI authors
were also instructed not to advertise
their work. ASE desk-rejected submissions that listed author information
on the first page, but not those that inadvertently revealed such information
in the text. Authors of OOPSLA submissions who revealed author identities were instructed to remove the
identities, which they did, and no paper was desk-rejected for this reason.
PLDI desk-rejected submissions that
revealed author identities in any way.
The review forms included optional
questions about author identities, the
answers to which were only accessible
to the PC chairs. The questions asked
if the reviewer thought he or she knew
the identity of at least one author, and
if so, to make a guess and to select
what informed the guess. The data
considered here refers to the first submitted version of each review. For ASE,
author identities were revealed to reviewers immediately after submission
of an initial review; for OOPSLA, ahead
of the PC meeting; for PLDI, only for
accepted papers, after all acceptance
decisions were made.
Threats to validity. Reviewers
were urged to provide a guess if they
thought they knew an author. A lack
of a guess could signify not following those instructions. However, this
risk is small, for example, OOPSLA
PC members were allowed to opt out
uniformly and yet 83% of the PC members participated. Asking reviewers
if they could guess author identities
may have affected their behavior: they
may not have thought about it had they
not been asked. Data about reviewers’
confidence in guesses may affect our
conclusions. Reviewers could submit multiple guesses per paper and
be considered correct if at least one
guess matched, so making many uninformed guesses could be considered
correct, but we did not observe this
phenomenon. In a form of selection
bias, all conferences’ review processes
were chaired by—and this Viewpoint
is written by—researchers who support double-blind review.
Anonymization Effectiveness
For the three conferences, 70%–86% of
reviews were submitted without guesses, suggesting that reviewers typically
did not believe they knew or were not
concerned with who wrote most of the
papers they reviewed. Figure 1 summarizes the number of reviewers, papers,
and reviews processed by each conference, and the distributions of author
identity guesses.
When reviewers did guess, they
were more likely to be correct (ASE 72%
of guesses were correct, OOPSLA 85%,
and PLDI 74%). However, 75% of ASE,
50% of OOPSLA, and 44% of PLDI pa-
Figure 1. Papers, reviews, reviewers, and author guesses. Reviewers include those on
the program and external committees, but exclude chairs. All papers received at least
three reviews; review load was non-uniform.
ASE OOPSLA PLDI
Reviewers 79 37 111
Papersaccepted 71 52 48
Papersrejected 263 144 240
Reviews 1,029 636 1,154
Did not contain a correct author guess 90.2% 74.4% 81.0%
Did not contain an author guess 86.4% 70.0% 74.3%
Tried to guess at least one author 14.7% 30.0% 25.7%
Guessed at least
one author correctly 9.8% 25.6% 19.1%
All author guesses incorrect 3.8% 4.4% 6.7%
Reviews with a guess 140 191 297
Guess at least one author correctly 72.1% 85.3% 74.1%
Guess all authors incorrectly 27.9% 14.7% 25.9%
Papersreviewed 334 196 288
No one tried guessing authors 66.5% 41.8% 40.6%
Someone guessed an author correctly 24.6% 50.0% 44.1%
All guesses incorrect 9.0% 8.2% 15.3%
Figure 2. Guess rate, and correct guess rate, by self-reported reviewer expertise score
(X: expert, Y: knowledgeable, Z: informed outsider).
ASE OOPSLA PLDI
Guess Correct Guess Correct Guess Correct
X 19.0% 74.7% 33.6% 86.7% 33.7% 74.2%
Y 11.2% 71.2% 29.3% 84.3% 24.6% 69.0%
Z 7.1% 55.6% 21.2% 83.3% 19.7% 48.6%