be very difficult to get what you really
need, which is the ability to drill down
into the details of why a given response
time is as large as it is.
Unfortunately, people tend to measure what is easy to measure, which
is not necessarily what they should be
measuring. It’s a bug. Measures that
aren’t what you need, but that are easy
enough to obtain and seem related to
what you need are called surrogate measures. Examples include subroutine call
counts and samples of subroutine call
I’m ashamed that I do not have
greater command over my native language than to say it this way, but here is
a catchy, modern way to express what I
think about surrogate measures:
surrogate measures suck.
Here, unfortunately, suck doesn’t
mean never work. It would actually
be better if surrogate measures never
worked. Then nobody would use them.
The problem is that surrogate measures
work sometimes. This inspires people’s
confidence that the measures they are
using should work all the time, and
then they don’t. Surrogate measures
have two big problems.
˲ ˲ They can tell you your system’s OK
when it is not. That’s what statisticians
call type I error, the false positive.
˲ ˲ They can tell you that something is
a problem when it is not. That’s a type
II error, the false negative. I have seen
each type of error waste years of people’s time.
When the time comes to assess the
specifics of a real system, your success
is at the mercy of how good the measurements are that your system allows
you to obtain. I have been fortunate to
work in the Oracle market segment,
where the software vendor at the center
of our universe participates actively in
making it possible to measure systems
the right way. Getting application software developers to use the tools that Oracle offers is another story, but at least
the capabilities are there in the product.
Performance is a feature
Performance is a software application
feature, just like recognizing that it’s
convenient for a string of the form “Case
1234” to automatically hyperlink over to
case 1234 in your bug-tracking system.
(FogBugz, which is software that I enjoy
using, does this.) Performance, like any
other feature, does not just happen; it
has to be designed and built. To do per-
formance well, you have to think about
it, study it, write extra code for it, test it,
and support it.
Thank you, Baron Schwartz for the
email conversation in which you
thought I was helping you, but in actual
fact, you were helping me come to grips
with the need for introducing coherency delay more prominently into my
thinking. Thank you, Jeff Holt, Ron Cris-co, Ken Ferlita, and Harold Palacio for
the daily work that keeps the company
going and for the lunchtime conversations that keep my imagination going.
Thank you, Tom Kyte for your continued
inspiration and support. Thank you,
Mark Farnham for your helpful suggestions. And thank you, Neil Gunther for
your patience and generosity in our ongoing discussions about knees.
You’re Doing It Wrong
hidden in Plain Sight
1. cMg (computer Measurement group, a network of
professionals who study these problems very, very
2. eight-second rule; http://en.wikipedia.org/wiki/
3. garvin, d. building a learning organization. Harvard
Business Review (July 1993).
4. gunther, n. universal Law of computational
scalability (1993); http://en.wikipedia.org/wiki/
5. Knuth, d. structured programming with go to
statements. ACM Computing Surveys 6, 4 (1974), 268.
6. Kyte, t. a couple of links and an advert…; http://tkyte.
7. Millsap, c. and holt, J. Optimizing Oracle Performance.
o’reilly, sebastopol, ca, 2003.
8. oak table network; http://www.oaktable.net.
Cary Millsap is the founder and president of Method r
corporation ( http://method-r.com), a company devoted to
software performance. he is the author (with Jeff holt) of
Optimizing Oracle Performance (o’reilly) and a co-author
of Oracle Insights: Tales of the Oak Table (apress). he is
the former vice president of oracle corporation’s system
Performance group and a co-founder of his former
company hotsos. he is also an oracle ace director and
a founding partner of the oak table network, an informal
association of well-known “oracle scientists.” Millsap
blogs at http://carymillsap.blogspot.com, and tweets at