growing in proportion to the number of
submissions and still promise presenters an influential audience, because
there are limits on the number of conferences that researchers can attend.
So attention by an ever-growing community necessarily remains focused on
a small set of conferences.
The high volume of submissions is
also triggering a second scaling problem: the shrinking pool of qualified
and willing PC candidates. The same
trends that are making the field exciting also bring all manner of opportunities to top researchers (who are highly
sought as PC members). Those who do
serve on PCs rightly complain that they
are overworked and unable to read all
˲ If submissions are read by only a
few PC members then there will be fewer broad discussions at PC meetings
about the most exciting new research
directions. Yet senior PC members often cite such dialogue as their main incentive for service.
˲If fewer senior researchers are
present at the PC meeting then serving
on the PC no longer provides informal
opportunities for younger PC members
to interact with senior ones.
And a growing sense that the process is broken has begun to reduce
We see a confluence
of factors that
the magnitude without
adding content to
a signal—the pool
the prestige associated with serving
on a PC. Service becomes more of a
burden and less likely to help in career
advancement. When serving on a PC
becomes unattractive, a sort of death
spiral is created.
In the past, journal publications
were mandatory for promotions at
the leading departments. Today, promotions can be justified with publications in top conferences (see, for
example, the CRA guidelines on tenureb). Yet conference publications are
shorter. This leads to more publications per researcher and per project,
even though the aggregate scientific
content of all these papers is likely
the same (albeit with repetition for
context-setting). So our current culture creates more units to review with
a lower density of new ideas.
Conference publications are an excellent way to alert the community to
a general line of inquiry or to publicize
an exciting recent result. Nevertheless,
we believe that journal papers remain
the better way to document significant
pieces of systems research. For one
thing, journals do not force the work
to be fractured into 12-page units. For
another, the review process, while potentially time consuming, often leads
to better science and a more useful
publication. Perhaps it is time for the
pendulum to swing back a bit.
Looking Back and Peering ahead
How did we get to this point? Historically, journals accepted longer papers
and imposed a process involving mul-
b See http://www.cra.org/reports/tenure_review.
tiple rounds of revision based on careful review. Publication decisions were
made by standing boards of editors,
who are independent and reflective.
So journal papers were justifiably perceived as archival, definitive publications. And thus they were required for
tenure and promotions.
This pattern shifted at least two decades ago, when the systems researchers themselves voted with their feet.
Given the choice between writing a
definitive journal paper about their
last system (having already published
a paper in a strong conference) versus building the next exciting system,
systems researchers usually opted to
build that next system. Computer science departments couldn’t face having their promising young leaders denied promotion over a lack of journal
publications, so they educated their
administrations about the unique culture of the systems area. With journal
publication no longer central to career
advancement, increasing numbers of
researchers chose the path offering
quicker turnaround, less dialogue with
reviewers, and that accepted smaller
contributions (which are easier to devise and document).
As submissions declined, journals
started to fill their pages by publishing
material from top conferences. Simultaneously, under cost pressure, journals limited paper lengths, undercutting one of their advantages. Reviewers
for journals receive little visibility or
thanks for their efforts, so it is a task
that often receives lower priority. And
that leads to publication delays that
some researchers argue make journal
publication unattractive, although
when ACM TOCS (a top systems jour-
nal) reduced reviewer delay, researchers remained resistant to submitting
Simultaneously, the top conferences have also evolved. Once, SOSP
and SIGCOMM were self-policed: submissions were not blinded, so submitting immature work to be read by
a program committee populated by
the field’s top researchers could tarnish your reputation. And the program
committees read all the submissions,
debating each acceptance decision
(and many rejections) as a group. An
c ACM Transactions on Computer Systems (TOCS).