Time pressure on reviewers has led
to non-reviews, which provide little to
no feedback. Sometimes, reviews are
sprinkled with words such as “
moronic,” “stupid,” or “myopic” that reflect
the frustration of reviewers. One solution might be to increase the size of
program committees by making them
more inclusive and to reduce overlap
in program committees. Reviewer bias
could be reduced by bringing back
double-blind reviews and by ensuring
that, as a general rule, reviewers do not
review papers they have earlier rejected
at another venue.
Almost a decade after the first incident, the editor who accused me of
misconduct sought me out and apologized. I thank this editor because his
apology allowed me to evaluate and
acknowledge the impact of that first
wrongful accusation. This second accusation has little impact on my career.
I am speaking up on behalf of young
researchers who are just embarking
on their careers. I appeal for rules and
guidelines, which protect David and
keep Goliath in check. I appeal for the
psychology of research to be incorporated into the review process, for program committees to have more diversity and less overlap, for reviewers to
understand their inherent biases, and
above all, for chairs and editors to have
a preponderance of evidence before
charging authors with wrongdoing.
After all, if Hardy had accused the unknown young mathematician who sent
him well-known theorems as original
work, one of the greatest mathematical geniuses, Ramanujan, would have
been lost to the world.
References
1. Publication Ethics by Council of Scientific Editors,
2016; http://www.councilscienceeditors.org/resource-library/editorial-policies/ white-paper-on-publication-ethics/
2. Kahneman, D. Thinking, Fast, and Slow. Farrar, Straus
and Giroux, 2011.
Elizabeth Varki ( varki@cs.unh.edu) is an associate
professor in the Department of Computer Science at the
University of New Hampshire.
I thank Rachelle Hollander for her guidance through
the review process and for steering this column toward
publication. I thank several anonymous reviewers for
their comments and suggestions, which have greatly
improved the column; special thanks to the reviewer
who suggested the column title. Finally, I thank Cindy
Childers, Larry Dowdy, Andras Fekete, Anu Mathai, and
Royce Withey for their support and feedback on early
versions of this column.
Copyright held by author.
authors believe that their paper is
correct, relevant, and original. Both
reviewers and authors may make mistakes, but errors/misjudgments by
reviewers can lead to punishment of
authors. Therefore, it is important for
editors/chairs to be impartial, which
is not possible without author input.
A reviewer spends a few hours on
a paper, while authors invest several
years, so authors often understand
their research more than the reviewer.
When a paper is rejected at one venue,
authors resubmit to another venue,
hoping that a new set of reviewers accept their paper. Authors try to revise
their paper based on reviewer comments, but it is sometimes impossible
to address all the reviewers’ concerns.
Possible reasons for not addressing
comments are: lack of time since the
next conference deadline follows immediately, author fatigue after going through countless revisions, lack
of resources to address comments,
contradictory comments by various
reviewers, to name a few. Therefore,
reviewers who are reassigned papers
they rejected elsewhere should not label authors as unethical if their comment is unaddressed in this new submission.
Sometimes rules are ambiguous and
authors unintentionally break a rule.
Apropos, rules on when and how to
cite one’s previous papers are contradictory: for double-blind, citing one’s
older papers is wrong; for single-blind, not citing one’s older papers
is wrong. When a paper is resubmitted, it is possible that authors forget
to add/remove their paper citation.
Sometimes, authors are simply embarrassed by their earlier paper and
choose not to cite it. Sometimes, authors do not cite older papers since it
appears as an attempt to increase citation count of their papers. Reviewers may attribute sinister intentions
where none exists. Instead of charging authors with misconduct, asking
authors for an explanation is reasonable. Editors/chairs may subsequently
choose to eliminate the paper from consideration without charging authors
with professional misconduct.
Reviewing is subject to error and
bias.
1, 2 In both incidents, the review-
ing was single-blind. The double-blind
review process is vanishing, which is
unfortunate. A reviewer might be bi-
ased by author names and affiliations,
so removal of the double-blind process
hurts researchers from lower-ranked
institutions. Moreover, some confer-
ences allow reviewers to resubmit their
review after seeing other submitted
reviews of the paper; this exacerbates
the problem of biased review. Authors
who are not part of the elite group have
to scale an impossibly high bar to get
their research published in reputed
conferences/journals.
In both incidents, charges were
brought by reviewers who had rejected
an earlier submission by the authors.
Before punishing us, the possibility of
reviewer fatigue and bias should have
been considered. A friend who is on
several program committees laughingly mentioned that he rejected a
paper, submitted to three different
conferences, thrice. The authors of
this paper may have given up without
realizing the paper was reviewed by
the same set of reviewers. This problem could be addressed by asking potential reviewers whether they have
previously reviewed (and rejected) the
paper, and if so, whether they could
impartially review this new submission. The framing of these questions
might help reviewers understand
their biases. If impartiality is not possible, then one should recuse oneself
from reviewing the paper.
Reviewing research papers is a difficult task, and I thank reviewers, editors,
program chairs, and others involved in
the process. It is arduous to read and
comprehend technical papers that are
likely written by young researchers
who are learning to articulate their research. In recent years, paper submissions have spiked, so reviewers may
be reading a large number of papers.
The review process
is unbalanced
with concentration
of power in the hands
of reviewers.