Communications - April 2009

ACM

Transactions on

Reconfigurable

Technology and

Systems

��

This quarterly publication is a peer-reviewed and archival journal that covers reconfigurable technology, systems, and applications on reconfigurable computers. Topics include all levels of reconfigurable system abstractions and all aspects of reconfigurable technology including platforms, programming environments and application successes.

��

www.acm.org/trets
www.acm.org/subscribe

part of the discipline’s attraction, but also complicates evaluation.

Across these variants, CS research exhibits distinctive characteristics, captured by seminal concepts: algorithm, computability, complexity, specification/implementation duality, recursion, fixpoint, scale, function/ data duality, static/dynamic duality, modeling, interaction…Not all scientists from other disciplines realize the existence of this corpus. Computer scientists are responsible for enforcing its role as basis for evaluation:

1. Computer science is an original discipline combining science and engineering. Researcher evaluation must be adapted to its specificity.

the cS Publication culture

In the computer science publication culture, prestigious conferences are a favorite tool for presenting original research—unlike disciplines where the prestige goes to journals and conferences are for raw initial results. Acceptance rates at selective CS conferences hover between 10% and 20%; in 2007–2008:

˲ ICSE (software engineering): 13% ˲ OOPSLA (object technology): 19% ˲ POPL (programming languages): 18%

Journals have their role, often to publish deeper versions of papers already presented at conferences. While many researchers use this opportunity, others have a successful career based largely on conference papers. It is important not to use journals as the only yardsticks for computer scientists.

Books, which some disciplines do not consider important scientific contributions, can be a primary vehicle in CS. Asked to name the most influential publication ever, many computer

the research
management edifice
relies on assessment
of researchers by
researchers.

scientists will cite Knuth’s The Art of Computer Programming. Seminal concepts such as Design Patterns first became known through books.

2. A distinctive feature of CS publication is the importance of selective conferences and books. Journals do not necessarily carry more prestige.

Publications are not the only scientific contributions. Sometimes the best way to demonstrate value is through software or other artifacts. The Google success story involves a fixpoint algorithm: Page Rank, which determines the popularity of a Web page from the number of links to it. Before Google was commercial it was research, whose outcome included a paper on Page Rank and the Google site. The site had—beyond its future commercial value—a research value that the paper could not convey: demonstrating scalability. Had the authors continued as researchers and come up for evaluation, the software would have been as significant as the paper.

Assessing such contributions is delicate: a million downloads do not prove scientific value. Publication, with its peer review, provides more easily de-codable evaluation grids. In assessing CS and especially Systems research, however, publications do not suffice:

3. To assess impact, artifacts such as software can be as important as publications.

Another issue is assessing individual contributions to multi-author work. Disciplines have different practices (2007–2008):

˲ Nature over a year: maximum coauthors per article 22, average 7. 3

˲ American Mathematical Monthly: 6, 2

˲ OOSPLA and POPL: 7, 2. 7

Disciplines where many coauthors are the norm use elaborate name-or-dering conventions to reflect individual contributions. This is not the standard culture in CS (except for such common practices as listing a Ph.D. student first in a joint paper with the advisor.

4. The order in which a CS publication lists authors is generally not significant. In the absence of specific indications, it should not serve as a factor in researcher evaluation.