tion databases that do not adequately
cover CS, such as Thomson Scientific’s
ISI Web of Science.
The principal problem is what ISI
counts. Many CS conferences and most
books are not listed; conversely, some
publications are included indiscriminately. The results make computer
scientists cringe.d Niklaus Wirth, Turing Award winner, appears for minor
papers from indexed publications,
not his seminal 1970 Pascal report.
Knuth’s milestone book series, with an
astounding 15,000 citations in Google
Scholar, does not figure. Neither do
Knuth’s three articles most frequently
cited according to Google.
Evidence of ISI’s shortcomings for
CS is “internal coverage”: the percentage of citations of a publication in the
same database. ISI’s internal coverage, over 80% for physics or chemistry,
is only 38% for CS.
Another example is Springer’s
Lecture Notes in Computer Science, which
ISI classified until 2006 as a journal.
A great resource, LNCS provides fast
publication of conference proceedings and reports. Lumping all into a
single “journal” category was absurd,
especially since ISI omits top non-LNCS conferences:
˲ The International Conference on
Software Engineering (ICSE), the top
conference in a field that has its own
ISI category, is not indexed.
˲ An LNCS-published workshop at
ICSE, where authors would typically
try out ideas not yet ready for ICSE
submission, was indexed.
ISI indexes SIGPLAN Notices, an
unrefereed publication devoting ordinary issues to notes and letters and
special issues to proceedings of such
conferences as POPL. POPL papers appear in ISI—on the same footing as a
reader’s note in a regular issue.
The database has little understanding of CS. Its 50 most cited CS references include “Chemometrics in food
science,” from a “Chemometrics and
Intelligent Laboratory Systems” journal. Many CS entries are not recognizable as milestone contributions. The
cruelest comparison is with CiteSeer,
whose Most Cited list includes many
publications familiar to all computer
scientists; it has not a single entry in
d All ISI searches as of mid-2008.
our focus is
common with the ISI list.
ISI’s “highly cited researchers” list
includes many prestigious computer
scientists but leaves out such iconic
names as Wirth, Parnas, Knuth and all
the 10 2000–2006 Turing Award winners
except one. Since ISI’s process provides
no clear role for community assessment,
the situation is unlikely to improve.
The inevitable deficiencies of alternatives pale in consideration:
9. In assessing publications and citations, ISI Web of Science is inadequate for
most of CS and must not be used. Alternatives include Google Scholar, CiteSeer,
and (potentially) ACM’s Digital Library.
Anyone in charge of assessment
should know that attempts to use ISI
for CS will cause massive opposition
and may lead to outright rejection
of any numerical criteria, including
more reasonable ones.
A recent trend is to rely on numerical measures of impact, derived from
citation databases, especially the
h-index, the highest n such that C (n) ≥
n, where C (n) is the citation count of
the author’s n-th ranked publication.
˲ The individual h-index divides the
h-index by the number of authors, better reflecting individual contributions.
˲ The g-index, highest n suchthatthe
top n publications received (together)
at least n2 citations, corrects another
h-index deficiency: not recognizing
extremely influential publications. (If
your second most cited work has 100
citations, the h-index does not care
whether the first has 101 or 15,000.)
The “Publish or Perish” sitee com-
e See http://www.harzing.com/resources.htm#/
putes these indexes from Google Scholar data. Such indexes cannot be more
credible than the underlying databases; results should always be checked
manually for context and possible distortions. It would be as counterproductive to reject these techniques as to use
them blindly to get definitive researcher assessments. There is no substitute
for a careful process involving complementary sources such as peer review.
Scientists are taught rigor: submit
any hypothesis to scrutiny, any experiment to duplication, any theorem to
independent proof. They naturally
assume that processes affecting their
careers will be subjected to similar
standards. Just as they do not expect,
in arguing with a Ph.D. student, to impose a scientifically flawed view on the
sole basis of seniority, so will they not
let management impose a flawed evaluation mechanism on the sole basis of
10. Assessment criteria must themselves undergo assessment and revision.
Openness and self-improvement
are the price to pay to ensure a successful process, endorsed by the community. This observation is representative
of our more general conclusion. Negative reactions to new assessment techniques deserve consideration. They
are not rejections of assessment per
se but calls for a professional, rational
approach. The bad news is that there is
no easy formula; no tool will deliver a
magic number defining the measure of
a researcher. The good news is that we
have ever more instruments at our disposal, which taken together can help
form a truthful picture of CS research
effectiveness. Their use should undergo the same scrutiny that we apply to
our work as scientists.
Bertrand Meyer ( firstname.lastname@example.org) is
a professor of software engineering at eth Zurich,
the swiss federal institute of technology, and chief
architect of eiffel software, santa barbara, ca.
Christine Choppy ( email@example.com.
fr) is a professor of computer science at université Paris
xiii and member of liPn (laboratoire d’informatique
de Paris nord), france.
Jørgen Staunstrup ( firstname.lastname@example.org) is provost of the it
university in copenhagen, Denmark.
Jan van Leeuwen ( email@example.com) is a professor of
computer science at utrecht university, the netherlands.
copyright held by author.
34 communicAtionS of the Acm | APriL 2009 | voL. 52 | no. 4