these tools, sort of in the way that early
humans learned to cook flesh. Let’s
just say that though the results may
have been edible, they were not winning any Michelin stars.
Using a packet-capture tool is, to
a networking person, somewhat like
using a thermometer is to a parent. It
is likely that if you ever felt sick when
you were a child at least one of your
parents would take your temperature.
If they took you to the doctor, the doctor would also take your temperature.
I once had my temperature taken for a
broken ankle—crazy, yes, but that doctor gave the best prescriptions, so I just
smiled blithely and let him have his
fun. That aside, taking a child’s temperature is the first thing on a parent’s
checklist for the question “Is my child
sick?” What on earth does this have to
do with capturing packets?
By far the best tool for determining
what is wrong with programs that use
a network, or even the network itself, is
the tcpdump tool. Why is that? Surely
in the now 40-plus years since packets were first transmitted across the
original ARPANET we have developed
some better tools. The fact is we have
not. When something in the network
breaks, you want to be able to see the
messages at as many layers as possible.
The other key component in debugging network problems is understanding the timing of what happens, which
a good packet-capture program also
records. Networks are perhaps the
most nondeterministic components
of any complex computing system.
Finding out who did what to whom
and when (another question parents
often ask, usually after a fight among
siblings) is extremely important.
All network protocols, and the programs that use them, have some sort
of ordering that is important to their
functioning. Did a message go missing? Did two or more messages arrive
out of order at the destination? All of
these questions can potentially be
answered by using a packet sniffer to
record network traffic, but only if you
It’s also important to record the network traffic as soon as you see the problem. Because of their nondeterministic
nature, networks give rise to the worst
types of timing bugs. Perhaps the bug
happens only every so many hours, be-
using a packet-
capture tool is,
to a networking
like using a
to a parent.
cause of a rollover in a large counter;
you really want to start recording the
network traffic before the bug occurs,
not after, because it may be many hours
until the condition comes up again.
So, here are some very basic recommendations on using a packet sniffer
in debugging a network problem. First,
get permission (yes, it really is KV giving you this advice). People get cranky
if you record their network traffic, such
as instant messages, email, and banking transactions, and then post it to a
mailing list. Just because some person
in IT was dumb enough to give you root
or admin rights on your desktop does
not mean you should just record everything and send it off.
Next, record only as much information as you need to debug the problem.
If you’re new at this you’ll probably
have the program suck up every packet
so you don’t miss anything, but that’s
problematic for two reasons: the first
is the previously mentioned privacy issue; and the second is that if you record
too much data, finding the bug will be
like finding a needle in a haystack—
only you’ve never seen a haystack that big.
Recording an hour of Ethernet traffic
on your LAN can capture a few hundred
million packets. No matter how good a
tool you have, it’s going to do a much
better job at finding a bug if you narrow
down the search.
If you do record a lot of data, don’t
try to share it all as one huge chunk.
See how these points follow each other? Most packet-capture programs
have options to say, “Once the capture file is full, close it and start a new
one.” Limiting files to one megabyte is
a nice start.
Finally, do not record your data on a
network file system. There is no better
way to ruin a whole set of packet-capture files than by having them capture
So there you have it: a brief introduction to capturing data so you can
debug a networking problem. Perhaps
now you can get yelled at on a mailing
list for something more egregious than
not taking your network’s temperature
before calling the doctor.
Debugging in an Asynchronous World
Kode Vicious Bugs Out
A Conversation with Steve Bourne, Eric
Allman, and Bryan Cantrill
I would like to dedicate this column
to my first editor, Mrs. B. Neville-Neil,
who passed away after a sudden illness
on December 9th, 2009; she was 65
My mother took language, both written and spoken, very seriously. The last
thing I wanted to hear upon showing
her an essay I was writing for school
was, “Bring me the red pen.” In those
days I did not have a computer; all my
assignments were written longhand
or on a typewriter and so the red pen
meant a total rewrite. She was a tough
editor, but it was impossible to question
the quality of her work or the passion
that she brought to the writing process.
All of the things Strunk and White have
taught others throughout the years my
mother taught me, on her own, with the
benefit of only a high school education
and a voracious appetite for reading.
It is, in large part, due to my mother’s influence that I am a writer today.
It is also due to her influence that I review articles, books, and code on paper, using a red pen. Her edits and her
unswerving belief that I could always
improve are, already, keenly missed.
—George Vernon Neville-Neil III