review articles
I
L
L
U
S
T
R
A
T
I
O
N
B
Y
S
P
O
O
K
Y
P
O
O
K
A
A
T
D
E
B
U
T
A
R
T
PEER REVIEW IS the process by which experts in some
discipline comment on the quality of the works of
others in that discipline. Peer review of written works
is firmly embedded in current academic research
practice where it is positioned as the gateway process
and quality control mechanism for submissions to
conferences, journals, and funding bodies across
a wide range of disciplines. It is probably safe to
assume that peer review in some form will remain a
cornerstone of academic practice for years to come,
evidence-based criticisms of this process in computer
science22, 32, 45 and other disciplines23, 28 notwithstanding.
While parts of the academic peer review process
have been streamlined in the last few decades to take
technological advances into account, there are many
more opportunities for computational
support that are not currently being
exploited. The aim of this article is to
identify such opportunities and de-
scribe a few early solutions for auto-
mating key stages in the established
academic peer review process. When
developing these solutions we have
found it useful to build on our back-
ground in machine learning and ar-
tificial intelligence: in particular, we
utilize a feature-based perspective in
which the handcrafted features on
which conventional peer review usu-
ally depends (for example, keywords)
can be improved by feature weight-
ing, selection, and construction (see
Flach17 for a broader perspective on
the role and importance of features in
machine learning).
Twenty-five years ago, at the start
of our academic careers, submitting a
paper to a conference was a fairly involved and time-consuming process
that roughly went as follows: Once an
author had produced the manuscript
(in the original sense, that is, manually produced on a typewriter, possibly
by someone from the university’s pool
of typists), he or she would make up to
seven photocopies, stick all of them
Computational
Support for
Academic
Peer Review:
A Perspective from
Artificial Intelligence
DOI: 10.1145/2979672
New tools tackle an age-old practice.
BY SIMON PRICE AND PETER A. FLACH
key insights
˽ State-of-the-art tools from machine
learning and artificial intelligence
are making inroads to automate parts
of the peer-review process; however,
many opportunities for further
improvement remain.
˽ Profiling, matching, and open-world
expert finding are key tasks that can
be addressed using feature-based
representations commonly used in
machine learning.
˽ Such streamlining tools also offer
perspectives on how the peer-review
process might be improved: in particular,
the idea of profiling naturally leads to
a view of peer review being aimed at
finding the best publication venue (if any)
for a submitted paper.
˽ Creating a more global embedding for
the peer-review process that transcends
individual conferences or conference
series by means of persistent reviewer
and author profiles is key, in our opinion,
to a more robust and less arbitrary
peer-review process.
