By mining high-quality bibliographic metadata from sources like PubMed, Profiles RNS infers implicit networks based on keywords, co-authors, department, location, and similar research. Researchers can also define their own explicit networks and curate their list of keywords and publications. Profiles RNS supports expert finding via a rich set of searching and browsing functions for traversing these networks. Profiles RNS is a noteworthy open source example of a growing body of research intelligence tools that compete to provide definitive databases of academics that, while varying in scope, scale and features, collectively constitute a valuable resource for a program chair seeking new reviewers. Well-known examples include free sites like academia.edu, Google Scholar, Mendeley, Microsoft Academic Search, ResearchGate, and numerous others that mine public data or solicit data directly from researchers themselves, as well as pay-to-use offerings like Elsevier’s Reviewer Finder.

Data issues. There is a wealth of publicly available data about the expertise of researchers that could, in principle, be used to profile program committee members (without requiring them to choose keywords or upload papers) or to suggest a ranked list of candidate invitees for any given set of topics. Obvious data sources include academic home pages, online bibliographies, grant awards, job titles, research group membership, events attended as well as membership of professional bodies and other reviewer pools. Despite the availability of such data, there are a number of problems in using it for the purpose of finding an expert on a particular topic.

If the data is to be located and used
automatically then it is necessary to
identify the individual or individuals
described by the data. Unfortunately
a person’s name is not guaranteed
to be a unique identifier (UID): of-
ten not being globally unique in the
first place, they can also be changed
through title, choice, marriage, and
so on. Matters are made worse be-
cause many academic reference styles
use abbreviated forms of a name us-
ing initials. International variations
in word ordering, character sets, and
alternative spellings make name
resolution even more challenging
for a peer review tool. Indeed, the
problem of author disambiguation is
sufficiently challenging to have mer-
ited the investment of considerable
research effort over the years, which
has in turn led to practical tool de-
velopment in areas with similar re-
quirements to finding potential peer
reviewers. For instance, Profiles RNS
supports finding researchers with
specific expertise and includes an Au-
thor Disambiguation Engine using
factors such as name permutations,
email address, institution affiliations,
known co-authors, journal titles, sub-
ject areas, and keywords.
To address these problems in their
own record systems, publishers and
bibliographic databases like DBLP
and Google Scholar have developed
their own proprietary UID schemes
for identifying contributors to pub-
lished works. However, there is now
considerable momentum behind the
non-proprietary Open Researcher and
Contributor ID (ORCID)e and publish-
ers are increasingly mapping their
own UIDs onto ORCID UIDs. A subtle
problem remains for peer review tools
when associating data, particularly
academic publications, with an indi-
vidual researcher because a great deal
of academic work is attributed to mul-
tiple contributors. Hope for resolving
individual contributions comes from a
concerted effort to better document all
outputs of research, including not only
papers but also websites, datasets, and
software, through richer metadata de-
scriptions of Research Objects. 10

Balance and coverage. Finding candidate reviewers is only part of a program chair’s task in forming a committee—attention must also be paid to coverage and balance. It is important to ensure more popular areas get proportionately more coverage than less popular ones while also not excluding less well known but potentially important new areas. Thus, there is a subjective element to balance and coverage that is not entirely captured by the score matrix. Recent work seeks to address this for conferences by refining clusters, computed from a score matrix, using a form of crowdsourcing from the program committee and from the

 

e http://orcid.org

We suggest a good
way is to think of
a reviewer’s job
to “profile” the
paper in terms of
its strong and weak
points, and separate
the reviewing job
proper from the
eventual accept/
reject decision.