By mining high-quality bibliographic
metadata from sources like PubMed,
Profiles RNS infers implicit networks
based on keywords, co-authors, department, location, and similar research. Researchers can also define
their own explicit networks and curate their list of keywords and publications. Profiles RNS supports expert
finding via a rich set of searching and
browsing functions for traversing
these networks. Profiles RNS is a noteworthy open source example of a growing body of research intelligence tools
that compete to provide definitive databases of academics that, while varying in scope, scale and features, collectively constitute a valuable resource for
a program chair seeking new reviewers. Well-known examples include free
sites like academia.edu, Google Scholar, Mendeley, Microsoft Academic
Search, ResearchGate, and numerous
others that mine public data or solicit
data directly from researchers themselves, as well as pay-to-use offerings
like Elsevier’s Reviewer Finder.
Data issues. There is a wealth of
publicly available data about the expertise of researchers that could, in
principle, be used to profile program
committee members (without requiring them to choose keywords or upload papers) or to suggest a ranked
list of candidate invitees for any given
set of topics. Obvious data sources
include academic home pages, online bibliographies, grant awards, job
titles, research group membership,
events attended as well as membership of professional bodies and other
reviewer pools. Despite the availability of such data, there are a number of
problems in using it for the purpose of
finding an expert on a particular topic.
If the data is to be located and used
automatically then it is necessary to
identify the individual or individuals
described by the data. Unfortunately
a person’s name is not guaranteed
to be a unique identifier (UID): of-
ten not being globally unique in the
first place, they can also be changed
through title, choice, marriage, and
so on. Matters are made worse be-
cause many academic reference styles
use abbreviated forms of a name us-
ing initials. International variations
in word ordering, character sets, and
alternative spellings make name
resolution even more challenging
for a peer review tool. Indeed, the
problem of author disambiguation is
sufficiently challenging to have mer-
ited the investment of considerable
research effort over the years, which
has in turn led to practical tool de-
velopment in areas with similar re-
quirements to finding potential peer
reviewers. For instance, Profiles RNS
supports finding researchers with
specific expertise and includes an Au-
thor Disambiguation Engine using
factors such as name permutations,
email address, institution affiliations,
known co-authors, journal titles, sub-
ject areas, and keywords.
To address these problems in their
own record systems, publishers and
bibliographic databases like DBLP
and Google Scholar have developed
their own proprietary UID schemes
for identifying contributors to pub-
lished works. However, there is now
considerable momentum behind the
non-proprietary Open Researcher and
Contributor ID (ORCID)e and publish-
ers are increasingly mapping their
own UIDs onto ORCID UIDs. A subtle
problem remains for peer review tools
when associating data, particularly
academic publications, with an indi-
vidual researcher because a great deal
of academic work is attributed to mul-
tiple contributors. Hope for resolving
individual contributions comes from a
concerted effort to better document all
outputs of research, including not only
papers but also websites, datasets, and
software, through richer metadata de-
scriptions of Research Objects.
10
Balance and coverage. Finding candidate reviewers is only part of a program chair’s task in forming a committee—attention must also be paid to
coverage and balance. It is important to
ensure more popular areas get proportionately more coverage than less popular ones while also not excluding less
well known but potentially important
new areas. Thus, there is a subjective
element to balance and coverage that
is not entirely captured by the score
matrix. Recent work seeks to address
this for conferences by refining clusters, computed from a score matrix,
using a form of crowdsourcing from
the program committee and from the
e http://orcid.org
We suggest a good
way is to think of
a reviewer’s job
to “profile” the
paper in terms of
its strong and weak
points, and separate
the reviewing job
proper from the
eventual accept/
reject decision.