growing in proportion to the number of submissions and still promise presenters an influential audience, because there are limits on the number of conferences that researchers can attend. So attention by an ever-growing community necessarily remains focused on a small set of conferences.

The high volume of submissions is also triggering a second scaling problem: the shrinking pool of qualified and willing PC candidates. The same trends that are making the field exciting also bring all manner of opportunities to top researchers (who are highly sought as PC members). Those who do serve on PCs rightly complain that they are overworked and unable to read all the submissions.

˲ If submissions are read by only a few PC members then there will be fewer broad discussions at PC meetings about the most exciting new research directions. Yet senior PC members often cite such dialogue as their main incentive for service.

˲If fewer senior researchers are present at the PC meeting then serving on the PC no longer provides informal opportunities for younger PC members to interact with senior ones.

And a growing sense that the process is broken has begun to reduce

We see a confluence
of factors that
amplify—increasing
the magnitude without
adding content to
a signal—the pool
of submissions.

the prestige associated with serving on a PC. Service becomes more of a burden and less likely to help in career advancement. When serving on a PC becomes unattractive, a sort of death spiral is created.

In the past, journal publications were mandatory for promotions at the leading departments. Today, promotions can be justified with publications in top conferences (see, for example, the CRA guidelines on tenure^b). Yet conference publications are shorter. This leads to more publications per researcher and per project, even though the aggregate scientific content of all these papers is likely the same (albeit with repetition for context-setting). So our current culture creates more units to review with a lower density of new ideas.

Conference publications are an excellent way to alert the community to a general line of inquiry or to publicize an exciting recent result. Nevertheless, we believe that journal papers remain the better way to document significant pieces of systems research. For one thing, journals do not force the work to be fractured into 12-page units. For another, the review process, while potentially time consuming, often leads to better science and a more useful publication. Perhaps it is time for the pendulum to swing back a bit.

 

Looking Back and Peering ahead How did we get to this point? Historically, journals accepted longer papers and imposed a process involving mul-

b See http://www.cra.org/reports/tenure_review. html.

tiple rounds of revision based on careful review. Publication decisions were made by standing boards of editors, who are independent and reflective. So journal papers were justifiably perceived as archival, definitive publications. And thus they were required for tenure and promotions.

This pattern shifted at least two decades ago, when the systems researchers themselves voted with their feet. Given the choice between writing a definitive journal paper about their last system (having already published a paper in a strong conference) versus building the next exciting system, systems researchers usually opted to build that next system. Computer science departments couldn’t face having their promising young leaders denied promotion over a lack of journal publications, so they educated their administrations about the unique culture of the systems area. With journal publication no longer central to career advancement, increasing numbers of researchers chose the path offering quicker turnaround, less dialogue with reviewers, and that accepted smaller contributions (which are easier to devise and document).

As submissions declined, journals started to fill their pages by publishing material from top conferences. Simultaneously, under cost pressure, journals limited paper lengths, undercutting one of their advantages. Reviewers for journals receive little visibility or thanks for their efforts, so it is a task that often receives lower priority. And that leads to publication delays that some researchers argue make journal publication unattractive, although when ACM TOCS (a top systems jour- c

nal) reduced reviewer delay, researchers remained resistant to submitting papers there.

Simultaneously, the top conferences have also evolved. Once, SOSP and SIGCOMM were self-policed: submissions were not blinded, so submitting immature work to be read by a program committee populated by the field’s top researchers could tarnish your reputation. And the program committees read all the submissions, debating each acceptance decision (and many rejections) as a group. An

 

c ACM Transactions on Computer Systems (TOCS).