Follow us on Twitter at http://twitter.com/blogCACM
The Communications Web site, http://cacm.acm.org,
features more than a dozen bloggers in the BLOG@CACM
community. In each issue of Communications, we’ll publish
selected posts or excerpts.
However, most testers feel like
they’re helping ensure the product’s
success when, in fact, they’re missing that component altogether. Myers
notes this misguided focus, stating,
“You cannot test a program to guarantee that it is error-free.” Software
by its nature has an unlimited number of bugs. Boris Beizer said in
Software Testing Techniques (1995, https://
amzn.to/2N1yOhg): “The probability
of showing that the software works decreases as testing increases; that is,
the more you test, the likelier you are
to find a bug. Therefore, if your objective is to demonstrate a high probability of working, that objective is best
achieved by not testing at all!” Most
testers fail to understand this. They
tend to see it as a neatly packaged
product that only needs a bit of polishing to run smooth.
The real number can be limitless.
No matter how big or small, simple or
complex, old or new a product is, the
potential for bugs is astronomical.
Myers underscores this, arguing that
“it is impractical, often impossible,
to find all the errors in a program.”
Even with limitless time and funding,
testers cannot find all the bugs. Bill
Hetzel in his book The Complete Guide
to Software Testing (1993, https://
amzn.to/2m6f5BM) wrote, “We can-
not achieve 100% confidence no mat-
ter how much time and energy we put
into it!” William E. Lewis in his book
Software Testing and Continuous Qual-
ity Improvement (2009, https://amzn.
to/2KUUsXg) even calls this a “testing
paradox,” which has “two underlying
and contradictory objectives: to give
confidence that the product is work-
ing well and to uncover errors in the
software product before its delivery to
the customer.” If this is the case, then
what do you do?
There has to be a certain point
where testers stop looking for bugs.
Meyers points out that “one of the
most difficult questions to answer
when testing a program is determining when to stop, since there is no way
of knowing if the error just detected
is the last remaining error.” Finding
bugs motivates testers, and they’ll
keep looking for them. At some point,
you have to launch the product. But
what happens, though, if you launch a
product before you find all the bugs?
If you do that, then won’t you launch
it with bugs? Yes!
The Era of Hackers
May 30, 2018
You gather up your team of
testers. You throw them some money. You
give them a schedule. And then you sit
back and watch them go, tearing through
your product trying to break it. They find
bugs, report them, find more bugs, report
those too. Soon enough, they’ll leave you
with the perfect product ready to ship
without any worries and total customer
satisfaction. Right? Wrong.
In The Art of Software Testing (2011,
Myers explains that “testing is the process of executing a program with the
intent of finding errors.” Testing fails
because the intentions behind the task
are very often misplaced. Finding errors is not the same strategy as making
sure a product works. Instead of focusing on whether the product functions
under parameters, the main focus of a
testing team must center on discovering bugs. This “destructive, even sadistic, process,” as Myers calls it, focuses
on breaking the product, looking for
various inputs to crash under stress.
or Ensuring Success?
Finding errors is not the same as making certain
a software product works correctly.
DOI: 10.1145/3237196 http://cacm.acm.org/blogs/blog-cacm