Time pressure on reviewers has led to non-reviews, which provide little to no feedback. Sometimes, reviews are sprinkled with words such as “ moronic,” “stupid,” or “myopic” that reflect the frustration of reviewers. One solution might be to increase the size of program committees by making them more inclusive and to reduce overlap in program committees. Reviewer bias could be reduced by bringing back double-blind reviews and by ensuring that, as a general rule, reviewers do not review papers they have earlier rejected at another venue.

Almost a decade after the first incident, the editor who accused me of misconduct sought me out and apologized. I thank this editor because his apology allowed me to evaluate and acknowledge the impact of that first wrongful accusation. This second accusation has little impact on my career. I am speaking up on behalf of young researchers who are just embarking on their careers. I appeal for rules and guidelines, which protect David and keep Goliath in check. I appeal for the psychology of research to be incorporated into the review process, for program committees to have more diversity and less overlap, for reviewers to understand their inherent biases, and above all, for chairs and editors to have a preponderance of evidence before charging authors with wrongdoing. After all, if Hardy had accused the unknown young mathematician who sent him well-known theorems as original work, one of the greatest mathematical geniuses, Ramanujan, would have been lost to the world.

 

References

1. Publication Ethics by Council of Scientific Editors,

2016; http://www.councilscienceeditors.org/resource-library/editorial-policies/ white-paper-on-publication-ethics/

2. Kahneman, D. Thinking, Fast, and Slow. Farrar, Straus and Giroux, 2011.

 

Elizabeth Varki ( [e-mail]) is an associate professor in the Department of Computer Science at the University of New Hampshire.

I thank Rachelle Hollander for her guidance through the review process and for steering this column toward publication. I thank several anonymous reviewers for their comments and suggestions, which have greatly improved the column; special thanks to the reviewer who suggested the column title. Finally, I thank Cindy Childers, Larry Dowdy, Andras Fekete, Anu Mathai, and Royce Withey for their support and feedback on early versions of this column.

Copyright held by author.

authors believe that their paper is correct, relevant, and original. Both reviewers and authors may make mistakes, but errors/misjudgments by reviewers can lead to punishment of authors. Therefore, it is important for editors/chairs to be impartial, which is not possible without author input.

A reviewer spends a few hours on a paper, while authors invest several years, so authors often understand their research more than the reviewer. When a paper is rejected at one venue, authors resubmit to another venue, hoping that a new set of reviewers accept their paper. Authors try to revise their paper based on reviewer comments, but it is sometimes impossible to address all the reviewers’ concerns. Possible reasons for not addressing comments are: lack of time since the next conference deadline follows immediately, author fatigue after going through countless revisions, lack of resources to address comments, contradictory comments by various reviewers, to name a few. Therefore, reviewers who are reassigned papers they rejected elsewhere should not label authors as unethical if their comment is unaddressed in this new submission.

Sometimes rules are ambiguous and authors unintentionally break a rule. Apropos, rules on when and how to cite one’s previous papers are contradictory: for double-blind, citing one’s older papers is wrong; for single-blind, not citing one’s older papers is wrong. When a paper is resubmitted, it is possible that authors forget to add/remove their paper citation. Sometimes, authors are simply embarrassed by their earlier paper and choose not to cite it. Sometimes, authors do not cite older papers since it appears as an attempt to increase citation count of their papers. Reviewers may attribute sinister intentions where none exists. Instead of charging authors with misconduct, asking authors for an explanation is reasonable. Editors/chairs may subsequently choose to eliminate the paper from consideration without charging authors with professional misconduct.

Reviewing is subject to error and
bias. 1, 2 In both incidents, the review-
ing was single-blind. The double-blind
review process is vanishing, which is
unfortunate. A reviewer might be bi-
ased by author names and affiliations,
so removal of the double-blind process
hurts researchers from lower-ranked
institutions. Moreover, some confer-
ences allow reviewers to resubmit their
review after seeing other submitted
reviews of the paper; this exacerbates
the problem of biased review. Authors
who are not part of the elite group have
to scale an impossibly high bar to get
their research published in reputed
conferences/journals.

In both incidents, charges were brought by reviewers who had rejected an earlier submission by the authors. Before punishing us, the possibility of reviewer fatigue and bias should have been considered. A friend who is on several program committees laughingly mentioned that he rejected a paper, submitted to three different conferences, thrice. The authors of this paper may have given up without realizing the paper was reviewed by the same set of reviewers. This problem could be addressed by asking potential reviewers whether they have previously reviewed (and rejected) the paper, and if so, whether they could impartially review this new submission. The framing of these questions might help reviewers understand their biases. If impartiality is not possible, then one should recuse oneself from reviewing the paper.

Reviewing research papers is a difficult task, and I thank reviewers, editors,

program chairs, and others involved in the process. It is arduous to read and comprehend technical papers that are likely written by young researchers who are learning to articulate their research. In recent years, paper submissions have spiked, so reviewers may be reading a large number of papers. The review process is unbalanced with concentration of power in the hands of reviewers.