papers for which they have a conflict of interest (this can be represented by a separate binary conflict matrix Cr×p). As this problem is underspecified, we will assume that further information is available in the form of a score matrix Mr×p expressing for each paper-reviewer pair how well they are matched by means of a non-negative number (higher means a better match). The best allocation is then the one that maximizes the element-wise matrix product Σi j Ai j Mi j while satisfying all constraints. 44

This one-dimensional definition of ‘best’ does not guarantee the best set of reviewers if a paper covers multiple topics, for example, a paper on machine learning and optimization could be assigned three reviewers who are machine learning experts but none who are optimization experts. This shortcoming can be addressed by replacing R with the set Rc such that each c-tuple ∈ Rc represents a possible assignment of c reviewers. 24, 25, 42 Recent works add explicit constraints on topic coverage to incorporate multiple dimensions into the definition of best allocation. 26, 31, 40 Other types of constraints have also been considered, including geographical distribution and fairness of assignments, as have alternative constraint solver algorithms. 3, 19, 20, 43 The score matrix can come from different sources, possibly a combination. Here, we review three possible sources: feature-based matching, profile-based matching, and bidding.

Feature-based matching. To aid assigning submitted papers to reviewers a short list of subject keywords is often required by mainstream CMS tools as part of the submission process, either from a controlled vocabulary, such as the ACM Computing Classification System (CCS),a or as a free-text “ folkson-omy.” As well as collecting keywords for the submitted papers, taking the further step of also requesting subject keywords from the body of potential reviewers enables CMS tools to make a straightforward match between the papers and the reviewers based on a count of the number of keywords they have in common. For each paper the reviewers can then be ranked in order of the number of matching keywords.

 

a http://www.acm.org/about/class/ (The examples in this article refer to ACM’s 1998 CCS, which was recently updated.)

If the number of keywords associated with each paper and each reviewer is not fixed then the comparison may be normalized by the CMS to avoid overly favoring longer lists of keywords. If the overall vocabulary from which keywords are chosen is small then the concepts they represent will necessarily be broad and likely to result in more matches. Conversely, if the vocabulary is large, as in the case of free-text or the ACM CCS, then concepts represented will be finer grained but the number of matches is more likely to be small or even non-existent. Also, manually assigning keywords to define the subject of written material is inherently subjective. In the medical domain, where taxonomic classification schemes are commonplace, it has been demonstrated that different experts, or even the same expert over time, may be inconsistent in their choice of keywords. 6, 7

When a pair of keywords does not literally match, despite having been chosen to refer to the same underlying concept, one technique often used to improve matching is to also match their synonyms or syntactic variants— as defined in a thesaurus or dictionary of abbreviations, for example, treating ‘code inspection’ and ‘walkthrough’ as equivalent; likewise for ‘SVM’ and ‘ support vector machine’ or ‘λ-calculus’ and ‘lambda calculus.’ However, if such simple equivalence classes are not sufficient to capture important differences between subjects—for example, if the difference between ‘code inspection’ and ‘walk-through’ is significant—then an alternative technique is to exploit the hierarchical structure of a concept taxonomy in order to represent the distance between concepts. In this setting, a match can be based on the common ancestors of concepts—either counting the number of shared ancestors or computing some edge traversal distance between a pair of concepts, for example, the former ACM CCS concept ‘D. 1. 6 Logic Programming’ has ancestors ‘D. 1 Programming Techniques’ and ‘D. Software,’ both of which are shared by the concept ‘D. 1. 5 Object-oriented Programming’, meaning that D. 1. 5 and D. 1. 6 have a non-zero similarity because they have common ancestors.

Obtaining a useful representation
of concept similarity from a taxonomy
is challenging because the measures
tend to assume uniform coverage
of the concept space such that the
hierarchy is a balanced tree. The ap-
proach is further complicated as it is
common for certain concepts to ap-
pear at multiple places in a hierarchy,
that is, taxonomies may be graphs
rather than just trees, and conse-
quently there may be multiple paths
between a pair of concepts. The situ-
ation grows worse still if different tax-
onomies are used to describe the sub-
ject of written works from different
sources because a mapping between
the taxonomies is required. Thus, it
is not surprising that one of the most
common findings in the literature on
ontology engineering is that ontolo-
gies, including taxonomies, thesauri,
and dictionaries, are difficult to de-
velop, maintain, and use. 12
So, even with good CMS support,
keyword-based matching still requires
manual effort and subjective decisions
from authors, reviewers and, some-
times, ontology engineers. One useful
aspect of feature-based matching using
keywords is that it allows us to turn a het-
erogeneous matching problem (papers
against reviewers) into a homogeneous
one (paper keywords against reviewer
keywords). Such keywords are thus a
simple example of profiles that are used
to describe relevant entities (papers
and reviewers). Next, we take the idea of
profile-based matching a step further by
employing a more general notion of pro-
file that incorporates nonfeature-based
representations such as bags of words.

Automatic feature construction with profile-based matching. The main idea of profile-based matching is to automatically build representations of semantically relevant aspects of both papers and reviewers in order to facilitate construction of a score matrix. An obvious choice of such a representation for papers is as a weighted bag-of-words (see “The Vector Space Model” sidebar). We then need to build similar profiles of reviewers. For this purpose we can represent a reviewer by the collection of all their authored or co-authored papers, as indexed by some online repository such as DBLP29 or Google Scholar. This collection can be turned into a profile in several ways, including: build the profile from a single document or Web page containing the bibliographic details of the reviewer’s publications (see