We Gone Too Far?
Seeking a better understanding of computing through
a mixture of theory and appropriate experimental evidence.
I RECENTLY SUBMITTED a paper to a conference, and when I got the reviews back, I noticed a reviewer had asked me spe- cifically to take out the experiments (in this case, simulations) that
I had presented partly as motivation,
and partly to demonstrate my results.
The conference was, unsurprisingly, a
theory conference; according to the reviewer the experiments were not only
unnecessary, they detracted from the
Jeffrey Ullman’s Viewpoint in this
issue of Communications observes
that many computer science areas are
overly focused on experiments as validation for their work. I do not disagree;
he raises several great points. My own
experience on several program committees for systems conferences, as
well as an author for papers for such
conferences, is that experiments are,
with very rare exception, a de facto requirement. How can you know if it is
really a good idea, the thinking goes,
unless there are experiments to back
it up? Indeed, in some settings, even
simulations are called into question;
experiments should be done in real,
deployed systems, as simulations can
oversimplify what goes on in the real
world. As Ullman points out, there are
gaping problems with this framework.
In particular, we run the risk of losing
strong ideas because they do not fall