template two fundamental questions:
˲ What should be the nature of the review and revision process? How rigorous need it be for a given kind of publication venue? Should a dialogue involving referees’ reviews and authors’ revisions plus rebuttals be required for all publication venues or just journals? How should promotion committees treat publication venues—like conferences—where acceptance is highly competitive but the decision process is less deliberative and nobody scrutinizes final versions of papers to confirm that issues were satisfactorily resolved? How do we grow a science where the definitive publications for important research are neither detailed nor carefully checked?
˲ Should we continue to have high-quality, “must-attend” conferences, with the excitement, simultaneity, and ad hoc in-the-halls discussions that these bring? If we do, and they remain few in number, does it make sense for these to be structured as a series of plenary sessions in which (only) the very best work is presented? As an alternative, conferences could make much greater use of large poster sessions or “brief presentation” sessions, structured so that no credible submission is excluded (printing associated full papers in the proceedings). By offering authors an early path to visibility, could these kinds of steps reduce pressure?
a high-Level View: What must change (and What must not)
An important role—if not the role—of conferences and journals is to communicate research results: impact is the real metric. And in this we see some reason for hope, because a community seeking to maximize its impact would surely not pursue a strategy of publishing modest innovations rather than revolutionary ones. Force fields are needed to encourage researchers to maximize their impact, but creating these force fields will likely require changing our culture and values.
Another Viewpoint columnd in this magazine suggested a game-based formulation of the situation, where the
d J. Crowcroft, S. Keshav, and N. McKeown. Scaling the academic publication process to Internet scale. Commun. ACM 52, 1 (Jan. 2009), 27–30.
winning strategy is one that incentiv-izes both authors and program committees to behave in ways that remedy the problems discussed here. One can easily conjure other characterizations of the situation and other means of redress. But any solution must be broad and flexible, since systems research is far from a static enterprise. A solution must accommodate a field that is becoming more interdisciplinary in some areas and more specialized in others, challenging the very definition of “systems.” For example, the systems research community is starting to embrace studying corporate infrastructure components that (realistically) can only be investigated in highly exclusive proprietary settings—publication and validation of results now brings new challenges.
Nevertheless, some initial steps to solving the field’s problems are evident. Why not make a deliberate effort to evaluate accomplishments in terms of impact? To the extent that we are a field of professionals who advance in our careers (or stall) on the basis of rigorous peer reviews, such a shift could have a dramatic effect. We need to learn to filter CVs inflated by the phenomena discussed previously, and we need to publicize and apply appropriate standards in promotions, awards, and in who we perceive as our leaders.
Program committees need to adapt their behavior. Today, PCs are not only decision-making bodies for paper acceptances but they have turned into rapid-response reviewing services for
any and all. If authors of the bottom two-thirds of the submissions did not receive detailed reviews, then there would be less incentive for them to submit premature work. And even if they did submit poorly developed papers, the workload of the PC would be substantially decreased given the reduced reviewing load. If some sort of reviewing service is needed by the field (beyond asking one’s research peers for their feedback on a draft) rather than overloading our PCs, we should endeavor to create one—the Web, social networks, and ad hoc cooperative enterprises like Wikipedia surely can be adapted to facilitate such a service.
Finally, authors must revisit what they submit and where they submit it, being mindful of their obligation as scientists to help create an archival literature for the field. Early, unpolished work should be submitted to workshops or conference tracks specifically designed for cutting-edge but less validated results. Presentation of work at such a workshop should not preclude later submitting a refined paper to a conference. And publishing papers at a conference should not block submitting a definitive work on that topic for careful review and ultimate publication in an archival journal.
Absent such steps or others that a communitywide discussion might yield, we shall find ourselves standing on the toes of our predecessors rather than on their shoulders. And we shall become less effective at solving the important problems that lie ahead, as systems become critical in society. Older and larger fields, such as medicine and physics, long ago confronted and resolved similar challenges. We are a much younger discipline, and we can overcome those problems too.
Ken Birman ( ken@cs.cornell.edu) is the n. rama rao professor of Computer science in the department of Computer science at Cornell university, ithaca, ny.
Fred B. Schneider ( fbs@cs.cornell.edu) is the samuel b. eckert professor of Computer science in the department of Computer science at Cornell university, ithaca, ny.
We are grateful to three Communications reviewers for their comments on our original submission; Jon Crowcroft, robbert van renesse, and gün sirer also provided extremely helpful feedback on an early draft. We are also grateful to the organizers and attendees of the 2008 nsdi Workshop on organizing Workshops, Conferences and symposia (Wo WCs), at which many of the topics discussed in this Viewpoint were raised.
Copyright held by author.
References:
Archives