finding tool will uncover some weird
examples. Given enough coders,
you’ll see the same thing. The fol-
lowing utterances were culled from
trial meetings:
Upon seeing an error report saying
the following loop body was dead code
foo(i = 1; i < 0; i++)
... deadcode ...
“No, that’s a false positive; a loop ex-
ecutes at least once.”
For this memory corruption error
(32-bit machine)
int a[ 2], b;
memset(a, 0, 12);
“No, I meant to do that; they are next
to each other.”
For this use-after-free
free(foo);
foo->bar = ...;
“No, that’s OK; there is no malloc
call between the free and use.”
As a final example, a buffer overflow
checker flagged a bunch of errors of the
form
unsigned p[ 4];
...
p[ 4] = 1;
“No, ANSI lets you write 1 past the
end of the array.”
After heated argument, the program-
mer said, “We’ll have to agree to dis-
agree.” We could agree about the dis-
agreement, though we couldn’t quite
comprehend it. The (subtle?) interplay
between 0-based offsets and buffer siz-
es seems to come up every few months.
While programmers are not often
so egregiously mistaken, the general
trend holds; a not-understood bug
report is commonly labeled a false
positive, rather than spurring the programmer to delve deeper. The result?
We have completely abandoned some
analyses that might generate difficult-to-understand reports.
How to handle cluelessness. You cannot often argue with people who are
sufficiently confused about technical
matters; they think you are the one
who doesn’t get it. They also tend to get
emotional. Arguing reliably kills sales.
What to do? One trick is to try to organize a large meeting so their peers do
…it’s not
uncommon for
tool improvement
to be viewed
as “bad” or at
least a problem.
the work for you. The more people in
the room, the more likely there is someone very smart and respected and cares
(about bugs and about the given code),
can diagnose an error (to counter arguments it’s a false positive), has been
burned by a similar error, loses his/her
bonus for errors, or is in another group
(another potential sale).
Further, a larger results meeting
increases the probability that anyone
laid off at a later date attended it and
saw how your tool worked. True story:
A networking company agreed to buy
the Coverity product, and one week
later laid off 110 people (not because of
us). Good or bad? For the fired people
it clearly wasn’t a happy day. However,
it had a surprising result for us at a
business level; when these people were
hired at other companies some suggested bringing the tool in for a trial,
resulting in four sales.
What happens when you can’t fix
all the bugs? If you think bugs are bad
enough to buy a bug-finding tool, you
will fix them. Not quite. A rough heuristic is that fewer than 1,000 bugs, then
fix them. More? The baseline is to record the current bugs, don’t fix them
but do fix any new bugs. Many companies have independently come up with
this practice, which is more rational
than it seems. Having a lot of bugs usually requires a lot of code. Much of it
won’t have changed in a long time. A
reasonable, conservative heuristic is
if you haven’t touched code in years,
don’t modify it (even for a bug fix) to
avoid causing any breakage.
A surprising consequence is it’s not
uncommon for tool improvement to be
viewed as “bad” or at least a problem.
Pretend you are a manager. For anything
bad you can measure, you want it to diminish over time. This means you are
improving something and get a bonus.
You may not understand technical issues that well, and your boss certainly doesn’t understand them. Thus,
you want a simple graph that looks like
Figure 1; no manager gets a bonus for
Figure 2. Representative story: At company X, version 2. 4 of the tool found
approximately 2,400 errors, and over
time the company fixed about 1,200 of
them. Then it upgraded to version 3. 6.
Suddenly there were 3,600 errors. The
manager was furious for two reasons:
One, we “undid” all the work his people
