papers for which they have a conflict
of interest (this can be represented by
a separate binary conflict matrix Cr×p).
As this problem is underspecified, we
will assume that further information
is available in the form of a score matrix Mr×p expressing for each paper-reviewer pair how well they are matched
by means of a non-negative number
(higher means a better match). The best
allocation is then the one that maximizes
the element-wise matrix product Σi j Ai j Mi j
while satisfying all constraints.
44
This one-dimensional definition of
‘best’ does not guarantee the best set
of reviewers if a paper covers multiple
topics, for example, a paper on machine
learning and optimization could be assigned three reviewers who are machine
learning experts but none who are optimization experts. This shortcoming can
be addressed by replacing R with the
set Rc such that each c-tuple ∈ Rc represents a possible assignment of c reviewers.
24, 25, 42 Recent works add explicit constraints on topic coverage to incorporate
multiple dimensions into the definition
of best allocation.
26, 31, 40 Other types of
constraints have also been considered,
including geographical distribution and
fairness of assignments, as have alternative constraint solver algorithms.
3, 19, 20, 43
The score matrix can come from different sources, possibly a combination.
Here, we review three possible sources:
feature-based matching, profile-based
matching, and bidding.
Feature-based matching. To aid assigning submitted papers to reviewers
a short list of subject keywords is often
required by mainstream CMS tools as
part of the submission process, either
from a controlled vocabulary, such as
the ACM Computing Classification System (CCS),a or as a free-text “
folkson-omy.” As well as collecting keywords
for the submitted papers, taking the
further step of also requesting subject
keywords from the body of potential
reviewers enables CMS tools to make
a straightforward match between the
papers and the reviewers based on a
count of the number of keywords they
have in common. For each paper the
reviewers can then be ranked in order
of the number of matching keywords.
a http://www.acm.org/about/class/ (The examples in this article refer to ACM’s 1998 CCS,
which was recently updated.)
If the number of keywords associated with each paper and each reviewer
is not fixed then the comparison may
be normalized by the CMS to avoid
overly favoring longer lists of keywords.
If the overall vocabulary from which
keywords are chosen is small then the
concepts they represent will necessarily be broad and likely to result in more
matches. Conversely, if the vocabulary
is large, as in the case of free-text or the
ACM CCS, then concepts represented
will be finer grained but the number
of matches is more likely to be small or
even non-existent. Also, manually assigning keywords to define the subject
of written material is inherently subjective. In the medical domain, where
taxonomic classification schemes are
commonplace, it has been demonstrated that different experts, or even the
same expert over time, may be inconsistent in their choice of keywords.
6, 7
When a pair of keywords does not
literally match, despite having been
chosen to refer to the same underlying concept, one technique often used
to improve matching is to also match
their synonyms or syntactic variants—
as defined in a thesaurus or dictionary
of abbreviations, for example, treating
‘code inspection’ and ‘walkthrough’ as
equivalent; likewise for ‘SVM’ and ‘
support vector machine’ or ‘λ-calculus’ and
‘lambda calculus.’ However, if such
simple equivalence classes are not sufficient to capture important differences
between subjects—for example, if the
difference between ‘code inspection’
and ‘walk-through’ is significant—then
an alternative technique is to exploit
the hierarchical structure of a concept
taxonomy in order to represent the distance between concepts. In this setting,
a match can be based on the common
ancestors of concepts—either counting
the number of shared ancestors or computing some edge traversal distance between a pair of concepts, for example,
the former ACM CCS concept ‘D. 1. 6
Logic Programming’ has ancestors
‘D. 1 Programming Techniques’ and
‘D. Software,’ both of which are shared
by the concept ‘D. 1. 5 Object-oriented
Programming’, meaning that D. 1. 5 and
D. 1. 6 have a non-zero similarity because
they have common ancestors.
Obtaining a useful representation
of concept similarity from a taxonomy
is challenging because the measures
tend to assume uniform coverage
of the concept space such that the
hierarchy is a balanced tree. The ap-
proach is further complicated as it is
common for certain concepts to ap-
pear at multiple places in a hierarchy,
that is, taxonomies may be graphs
rather than just trees, and conse-
quently there may be multiple paths
between a pair of concepts. The situ-
ation grows worse still if different tax-
onomies are used to describe the sub-
ject of written works from different
sources because a mapping between
the taxonomies is required. Thus, it
is not surprising that one of the most
common findings in the literature on
ontology engineering is that ontolo-
gies, including taxonomies, thesauri,
and dictionaries, are difficult to de-
velop, maintain, and use.
12
So, even with good CMS support,
keyword-based matching still requires
manual effort and subjective decisions
from authors, reviewers and, some-
times, ontology engineers. One useful
aspect of feature-based matching using
keywords is that it allows us to turn a het-
erogeneous matching problem (papers
against reviewers) into a homogeneous
one (paper keywords against reviewer
keywords). Such keywords are thus a
simple example of profiles that are used
to describe relevant entities (papers
and reviewers). Next, we take the idea of
profile-based matching a step further by
employing a more general notion of pro-
file that incorporates nonfeature-based
representations such as bags of words.
Automatic feature construction
with profile-based matching. The main
idea of profile-based matching is to
automatically build representations of
semantically relevant aspects of both
papers and reviewers in order to facilitate construction of a score matrix. An
obvious choice of such a representation
for papers is as a weighted bag-of-words
(see “The Vector Space Model” sidebar).
We then need to build similar profiles of
reviewers. For this purpose we can represent a reviewer by the collection of all
their authored or co-authored papers,
as indexed by some online repository
such as DBLP29 or Google Scholar. This
collection can be turned into a profile
in several ways, including: build the
profile from a single document or Web
page containing the bibliographic details of the reviewer’s publications (see