recent issue of interactions magazine, I
have a keener appreciation for the possible
contribution of some technologies in
search of problems, but I still believe these
are more the exception than the norm …
and that without adequately making the
case for the human-centered value(s) the
systems will help realize, such papers are
probably more suitable for other venues.
One problem is that our field is moving
so fast that we have to allow new ideas to
cross evolve with other ideas rapidly. If we
require evaluations of every paper, then we
don’t have the rapid turnaround required for
innovations to cross paths with each other.
don’t agree on
what makes a good
how can we
to meet a standard
to keep in mind all the different ways a
systems paper can be really valuable and
worth seeing at a conference like CHI. A
systems paper could be interesting thanks
to a thorough analysis of its deployment
and usage (evaluation). Or it could be
interesting thanks to a well-argued
discussion of why it was built a particular
way (design). Or it might just demonstrate
that a given interesting capability could
be created at all. Or it could be a careful
argument about why a certain system
would be really useful, even if it hasn’t
been built or evaluated yet (motivation/
position paper). In the end, what I want are
papers that stimulate thought and action.
I’m not going to demand any particular
levels of motivation, design, or evaluation;
rather, I’m going to ask whether the
whole is innovative enough. This is a highly
subjective decision, which is why I greatly
value wise program committees who can
make such a judgment on my behalf.
I like your list, and think that the
bullet points are representative of good
evaluation criteria for systems papers
across computer science.
tessa Lau responds
Thank you all for the interesting discussion. My goal was to initiate a discussion within our community, not to
have all the answers.
Along those lines, Dan, the question
you raise about what constitutes “
appropriate evidence” is one that I’ll turn
back to the community for a collective
For what it’s worth, though, I don’t
think of your examples as “systems.”
The first is usage scenarios or design.
The second is an algorithm. The third
is an interaction method. Each of those
is fairly self-contained and possible to
evaluate using a fairly closed study.
What I meant by “systems” is an implemented prototype that gives people
access to new functionality that did not
exist before. Examples of “systems” include CoScripter, Many Eyes, Landay’s
DENIM and SILK, Gajos’s SUPPLE, Andrew Ko’s Whyline. How can we show
that each of these systems is “
innovative enough” (in David’s words) to merit publication?
Tessa Lau is a research staff member and manager at
IBm’s almaden research center.
© 2010 acm 0001-0782/10/1100 $10.00