tion databases that do not adequately cover CS, such as Thomson Scientific’s ISI Web of Science.

The principal problem is what ISI counts. Many CS conferences and most books are not listed; conversely, some publications are included indiscriminately. The results make computer scientists cringe.^d Niklaus Wirth, Turing Award winner, appears for minor papers from indexed publications, not his seminal 1970 Pascal report. Knuth’s milestone book series, with an astounding 15,000 citations in Google Scholar, does not figure. Neither do Knuth’s three articles most frequently cited according to Google.

Evidence of ISI’s shortcomings for CS is “internal coverage”: the percentage of citations of a publication in the same database. ISI’s internal coverage, over 80% for physics or chemistry, is only 38% for CS.

Another example is Springer’s Lecture Notes in Computer Science, which ISI classified until 2006 as a journal. A great resource, LNCS provides fast publication of conference proceedings and reports. Lumping all into a single “journal” category was absurd, especially since ISI omits top non-LNCS conferences:

˲ The International Conference on Software Engineering (ICSE), the top conference in a field that has its own ISI category, is not indexed.

˲ An LNCS-published workshop at ICSE, where authors would typically try out ideas not yet ready for ICSE submission, was indexed.

ISI indexes SIGPLAN Notices, an unrefereed publication devoting ordinary issues to notes and letters and special issues to proceedings of such conferences as POPL. POPL papers appear in ISI—on the same footing as a reader’s note in a regular issue.

The database has little understanding of CS. Its 50 most cited CS references include “Chemometrics in food science,” from a “Chemometrics and Intelligent Laboratory Systems” journal. Many CS entries are not recognizable as milestone contributions. The cruelest comparison is with CiteSeer, whose Most Cited list includes many publications familiar to all computer scientists; it has not a single entry in

d All ISI searches as of mid-2008.

our focus is
evaluation of
individuals rather
than departments
or laboratories.

common with the ISI list.

ISI’s “highly cited researchers” list includes many prestigious computer scientists but leaves out such iconic names as Wirth, Parnas, Knuth and all the 10 2000–2006 Turing Award winners except one. Since ISI’s process provides no clear role for community assessment, the situation is unlikely to improve.

The inevitable deficiencies of alternatives pale in consideration:

9. In assessing publications and citations, ISI Web of Science is inadequate for most of CS and must not be used. Alternatives include Google Scholar, CiteSeer, and (potentially) ACM’s Digital Library.

Anyone in charge of assessment should know that attempts to use ISI for CS will cause massive opposition and may lead to outright rejection of any numerical criteria, including more reasonable ones.

Assessment formulae

A recent trend is to rely on numerical measures of impact, derived from citation databases, especially the h-index, the highest n such that C (n) ≥ n, where C (n) is the citation count of the author’s n-th ranked publication. Variants exist:

˲ The individual h-index divides the h-index by the number of authors, better reflecting individual contributions.

˲ The g-index, highest n suchthatthe top n publications received (together) at least n² citations, corrects another h-index deficiency: not recognizing extremely influential publications. (If your second most cited work has 100 citations, the h-index does not care whether the first has 101 or 15,000.)

The “Publish or Perish” site^e com-

e See http://www.harzing.com/resources.htm#/ pop.htm.

putes these indexes from Google Scholar data. Such indexes cannot be more credible than the underlying databases; results should always be checked manually for context and possible distortions. It would be as counterproductive to reject these techniques as to use them blindly to get definitive researcher assessments. There is no substitute for a careful process involving complementary sources such as peer review.

Assessing Assessment

Scientists are taught rigor: submit any hypothesis to scrutiny, any experiment to duplication, any theorem to independent proof. They naturally assume that processes affecting their careers will be subjected to similar standards. Just as they do not expect, in arguing with a Ph.D. student, to impose a scientifically flawed view on the sole basis of seniority, so will they not let management impose a flawed evaluation mechanism on the sole basis of authority:

10. Assessment criteria must themselves undergo assessment and revision.

Openness and self-improvement are the price to pay to ensure a successful process, endorsed by the community. This observation is representative of our more general conclusion. Negative reactions to new assessment techniques deserve consideration. They are not rejections of assessment per se but calls for a professional, rational approach. The bad news is that there is no easy formula; no tool will deliver a magic number defining the measure of a researcher. The good news is that we have ever more instruments at our disposal, which taken together can help form a truthful picture of CS research effectiveness. Their use should undergo the same scrutiny that we apply to our work as scientists.

 

Bertrand Meyer ( bertrand.meyer@inf.ethz.ch) is a professor of software engineering at eth Zurich, the swiss federal institute of technology, and chief architect of eiffel software, santa barbara, ca.

Christine Choppy ( christine.choppy@lipn.univ-paris13. fr) is a professor of computer science at université Paris xiii and member of liPn (laboratoire d’informatique de Paris nord), france.

Jørgen Staunstrup ( jst@itu.dk) is provost of the it university in copenhagen, Denmark.

Jan van Leeuwen ( jan@cs.uu.nl) is a professor of computer science at utrecht university, the netherlands.

copyright held by author.

34 communicAtionS of the Acm | APriL 2009 | voL. 52 | no. 4