sures is high. Not all differences are
statistically significant at the level of
95%; some of our claims of nonsignificance may be revised if a larger sample
is used; for example, we were surprised
to find no statistically significant difference between the group of higher-productivity areas—ARCH, COMM,
DC, and IPCV—and the middle group,
including all areas but MIS and OR. A
larger sample size might reveal whether
this difference is significant.
Most research on bibliometric aspects of CS, including Biryukov and
Dong, 2 Elmacioglu and Lee, 4
Franceschet, 5 and Laender et al., 7 uses the
whole DBLP as its data source. We
could not follow such an approach.
As pointed out by Laender et al. 7 and
Feitz and Hoffmann, 11 DBLP has different coverage for different CS areas.
If we used DBLP as the data source we
would not know if the difference in
productivity was due to the different
practices of researchers in different areas or the difference in the DBLP coverage of these areas. We therefore used
DBLP to define the populations and
the set of researchers and publications
in each CS area, but the final productivity measurements were not based
on DBLP but on the papers listed on
researchers’ personal webpages. However, we used the DBLP data to define
the size of each area in order to correlate it with the citation rates.
The procedure we describe here is
repeatable. One may choose a different
set of areas and initial seeds to explore
more specific questions. The costly
step is defining the sample, or finding which researchers have up-to-date
webpages listing their publications.
For help expanding on our results or
exploring other issues regarding CS
publication and citation practices, we
provide the data used in this research
Our main purpose here is to pro-
vide the data needed to establish
evaluation criteria for CS researchers;
for example, one should not propose
a single journal productivity goal for
both COMM and AI researchers, as it
would be unfair to AI researchers and
could ultimately drive researchers
away from the area. Productivity and
citation rates differ between some but
not all CS areas, and evaluation crite-
ria for CS researchers must account
for these differences.
We are not able to claim one publica-
tion practice is “better” than another.
Moreover, it may not be possible for
a research community to change its
publication practices without under-
going internal turmoil, though cita-
tion practices may be more amenable
to change. Areas with low citation
rates may look to areas like DB and
GRAPH, which for most other charac-
teristics are in the mainstream of CS
practices but still have very high cita-
tion rates. It seems papers in these
areas dedicate much more space to
show how the research connects to
previously published papers, with a
corresponding increase in the refer-
ences they include. CS areas that may
be limited in their citation rates may
consider encouraging all authors, es-
pecially of conference papers, to in-
clude more elaborate analysis and in-
clusion of the related literature.
1. barbosa, S.D.J. and de Souza, c.S. are hcI
researchers an endangered species in brazil?
Interactions 18, 3 (may/June 2011), 69–71.
2. biryukov, m. and Dong, c. analysis of computer
science communities based on DblP. In Research
and Advanced Technology for Digital Libraries, Volume
6273 of LNCS. Springer, berlin 2010, 228–235.
3. chen, c. classification of scientific networks using
aggregated journal-journal citation relations in the
journal citation reports. Journal of the American
Society for Information Science and Technology 59, 14
(Dec. 2008), 2296–2304.
4. elmacioglu, e. and lee, D. on six degrees of separation
in DblP-Db and more. ACM SIGMOD Record 34, 2
(June 2005), 33–40.
5. franceschet, m. collaboration in computer science: a
network science approach. Journal of the American
Society for Information Science and Technology 62, 10
(oct. 2011), 1992–2012.
6. grudin, J. A List of Publication Culture Discussions and
Resources, 2012; http://research.microsoft.com/en-us/
7. laender, a., de lucena, c., maldonado, J., de Souza
e Silva, e., and zivianim, n. assessing the research
and education quality of the top brazilian computer
science graduate programs. ACM SIGCSE Bulletin 40,
2 (June 2008), 135–145.
8. lee, f., grijalva, t., and nowell, c. ranking economics
departments in a contested discipline: a bibliometric
approach to quality equality between theoretically
distinct subdisciplines. American Journal of Economics
and Sociology 69, 5 (nov. 2010), 1345–1375.
9. meyer, b., choppy, c., Staunstrup, J., and van leeuwen,
J. research evaluation for computer science.
Commun. ACM 52, 4 (apr. 2009), 31–34.
10. Patterson, D., Snyder, l., and ullman, J. Evaluating
Computer Scientists and Engineers for Promotion
and Tenure. technical report. computing research
association, Washington, D.c., aug. 1999; http://www.
11. reitz, f. and hoffmann, o. an analysis of the evolving
coverage of computer science subfields in the DblP
digital library. Research and Advanced Technology
for Digital Libraries. Volume 6273 of LNCS. Springer,
berlin, 2010, 216–227.
12. rosen-zvi, m., griffiths, t., Steyvers, m., and Smyth, P.
the author-topic model for authors and documents.
In Proceedings of the 20th Conference on Uncertainty
in Artificial Intelligence (banff, canada, July 7–11).
auaI Press, arlington, Va, 2004, 487–494.
13. Wagner, c., roessner, J., bobb, k., klein, J., boyack,
k., keyton, J., rafols, I., and börner, k. approaches
to understanding and measuring interdisciplinary
scientific research: a review of the literature. Journal
of Informetrics 5, 1 (Jan. 2011), 14–26.
14. Wainer, J., billa, c., and goldenstein, S. Invisible
work in standard bibliometric evaluation of computer
science. Commun. ACM 54, 5 (may 2011), 141–146.
15. Wainer, J., eckmann, m., goldenstein, S., and rocha,
a. Differences in Productivity and Impact Across
the Different Computer Science Subareas. Technical
Report IC-12-08. Institute of computing, university
of campinas, campinas, brazil, mar. 2012; http://www.
Jacques Wainer ( firstname.lastname@example.org) is a professor
in the computing Institute of the university of campinas,
Michael Eckmann ( email@example.com) is an
associate professor in the Department of mathematics
and computer Science of Skidmore college, Saratoga
Siome Goldenstein ( firstname.lastname@example.org) is an
associate professor in the computing Institute of the
university of campinas, campinas, brazil.
Anderson Rocha ( email@example.com) is an
assistant professor at the computing Institute at the
university of campinas, campinas, brazil.