That makes these tools more usable,
but it also means they will fail to report
real bugs. Dawson Engler and his col-
leagues made exactly this choice for
Coverity’s “unsound” static analyzer. 4
The state of the art in testing tools
has also advanced dramatically in the
past decade. Randomized fuzz testing
can be combined with static analysis
to explore paths that lead to failure.
These tools are now in the mainstream:
for example, Microsoft’s Driver Verifier
can test device-driver code for a variety
of problems and now includes ran-
domized concurrency stress testing.
As Edsger Dijkstra famously re-
marked, however, “Program testing
can be used to show the presence of
bugs, but never to show their absence!”
At some point, testing will fail to turn
up new bugs, which will unfortunately
photograph By hugh talMan, Courtesy of the sMIthsonIan InstItutIon
be discovered only after the software
Fixing Bugs: Risky
(and Slow) Business
Finding the bugs is only the first step.
Once a bug is found—whether by inspection, testing, or analysis—fixing it
remains a challenge. Any bug fix must
be undertaken with extreme care, since
any new code runs the risk of introducing yet more bugs. Developers must
construct and carefully test a patch to
ensure it fixes the bug without introducing any new ones. This can be costly and time-consuming. For example,
the average time between the discovery
of a remotely exploitable memory error
and the release of a patch for enterprise applications is 28 days, according
to Symantec. 12
At some point, fixing certain bugs
simply stops making economic sense.
Tracking their source is often difficult and time-consuming, even when
the full memory state and all inputs
to the program are available. Obviously, showstopper bugs must be fixed.
For other bugs, the benefits of fixing
them may be outweighed by the risks
of creating new bugs and the costs in
programmer time and delayed deployment.
Once the faulty software has been
deployed, the problem of chasing
down and repairing bugs becomes exponentially more difficult. Users rarely
provide detailed bug reports that allow
developers to reproduce the problem.
For deployed software on desktops
or mobile devices, getting enough
information to find a bug can be dif-