Communications - August 2012

accuracy makes them unsuitable for CS research evaluation.

Relative Relevant undercitation

To study the accuracy of database citation records, we measure the records’ relative relevant undercitation (RRU); the RRU of a database query is the fraction of all (cited, citing) paper pairs for which both cited and citing papers are indexed in the database but for which the database has no record of the citing paper in the cited-by list of the cited paper. This fraction equals the underestimation of the citation count reported by the database within its own coverage; in Figure 1, the citation of J1 by J2 is missing in WoS, but the citation by C is present, resulting in an RRU of 50%.

To compute RRUs, we developed a Python tool for querying six online databases—the ACM DL, CiteSeerX, DBLP, GS, Scopus, and WoS—by mimicking a researcher manually browsing a database by sending similar HTTP and parsing retrieved (HTML) data. Given a reference list of an author’s papers, the tool first queries the databases by title; for papers not found by title, it tries searching by cited author. The search is limited to the papers in the reference list to prevent counting publications by other authors with the same name or initials. 10

For each paper found in a data-
base, the tool retrieves its cited-by
list. In its extended mode of opera-
tion, it downloads the Bib TeX or End-
Note descriptions provided by the
database for all entries in that list. In
its fast mode the tool instead parses
the HTML pages to identify the citing
papers. As those pages display infor-
mation in a less-uniform way than
EndNote or Bib TeX, the fast mode can
produce less-accurate results. How-
ever, this mode is considerably faster
for most databases than its extended-
search mode, as fewer HTTP queries
are needed. Most databases try to de-
tect and block seemingly automated
querying. To work around this filter,
the tool is designed to sleep a random
amount of time, say, 25 to 35 seconds
between consecutive queries to GS.
The result of this first search phase
is a list of (cited, citing) paper pairs
of citations, each recorded by at least
one database. This list constitutes the
tool’s estimate of an author’s publica-
tion genuine citation count.

Experiments

We used the tool in 2010 and 2011
to perform three complementary ex-
periments: First, we set it to search all
aforementioned databases for three
authors, using its extended-search
mode. Though we focus here on cita-
tion accuracy, the experiment also en-
abled us to compare a database’s cov-
erage on the basis of what the three
authors would consider their own
relevant output. Due to the tool’s long
running times—several weeks—we
were able to study only three authors
in the experiment. We next searched
GS and WoS for 14 editors-in-chief
of various CS transactions published
by ACM and the IEEE Computer So-
ciety. Using the tool’s fast mode, we
thus limited searches to publication
lists we obtained from DBLP. The ex-
periment was less accurate and cov-
ered fewer databases than the first
experiment but included many more
authors and publications, enabling
us to validate the trends we observed
in the first experiment. Finally, we
performed a similar experiment for
GS and WoS for eight ACM and IEEE
transactions published from 2000 to
2002 to study the influence of RRU on
journal impact factors.

table 1. number of journal/conference papers in authors’ reference lists and in online publication databases.

Author (domain)
Koen de Bosschere
(compilers, computer architecture)
Bart dhoedt
(distributed computing, networks)
Wilfried Philips
(image and video processing)