OS X, and Linux are all very happy to run up to multigigabit speeds.

JacoBson: But the packet output isn’t driven by window scaling. Window scaling just gives you room to expand the window, but it expands only if the system defaults to a large window. For years, the default window was sized to fit the roughly 100-millisecond/1-mb- ps TCP paths you typically found back then. At some point over the past five or six years, people have decided that the common path ought to be considered a gigabit or more. As a consequence, the buffer size has been bumped up to more than a megabyte.

illustration by John balestrieri based on a PhotograPh courtesy of Van Jacobson

WeaVeR: Unfortunately, that’s almost exactly the case. If you look at my house, in fact, going from my computer to the file server, I’ve got a gig link at 10-plus milliseconds.

JacoBson: Have you done the experiment of varying the window size and then looking at throughput to your file server as a function of the window size to see whether or not you actually need the megabyte?

WeaVeR: No, I have not.

JacoBson: Well, I have, and the bandwidth flattops at an eight-packet window. I’m talking about a Linux file server running a very current kernel. Typically, the situation is that you’re in a home environment with large bandwidth and very short delays, or you’re going out over the Internet where you’re going to encounter small bandwidth and much longer delays—often

10 times or more longer than what you have at home. You could pick a window size that was appropriate for the Internet—something like 100 kilobytes, which is 10 kilobits into 100 milliseconds—and that would work more than adequately for your home environment as well.

But that just isn’t what we see anymore. Instead, we ship several megabytes, and that’s when you hear the complaint: “Oh my God, I’m seeing a second of delay.” Well, yes, of course. That’s how your system is configured.

Ge TTys: What’s more, you find that sort of thing in several different places. The operating system itself may have problems. If you look at the device drivers, you’ll find they’ve also grown some very large buffers of their own. Basically, whenever you look inside any of our current operating systems, you’ll find a recurring theme of bufferbloat at multiple layers.

That’s actually part of the problem: people are thinking of this as a system of layers, where they don’t have to worry about how everything actually works, either above them or below them. They aren’t really thinking through all the consequences of what they’re doing— and it shows!

Van JacoBson
We’ve put a lot
of effort into
protocol, router,
and algorithm
development that
makes it possible
for a single TcP
to saturate
a 40-gigabit link,
but we haven’t
put anything
remotely like that
effort into producing
usable links for
cellphone data,
DsL, or home cable.
That’s the hard
problem because
it requires you
think about how
the buffering
actually works
and how it turns
into latency.

If we do indeed face an impending crisis, what’s to be done? Bufferbloat presents a hard problem, with no single right solution. And making head-