BY ALL ACCOUNTS, today’s Internet is not moving data
as well as it should. Most of the world’s cellular users
experience delays of seconds to minutes; public Wi-Fi in
airports and conference venues is often worse. Physics
and climate researchers need to exchange petabytes of
data with global collaborators but find their carefully
engineered multi-Gbps infrastructure often delivers at
only a few Mbps over intercontinental distances. 6
These problems result from a design choice made
when TCP congestion control was created in the
1980s—interpreting packet loss as “congestion.” 13
This equivalence was true at the time but was because
of technology limitations, not first principles. As
NICs (network interface controllers) evolved from
Mbps to Gbps and memory chips from KB to GB, the
relationship between packet loss and congestion
became more tenuous.
Today TCP’s loss-based congestion control—even
with the current best of breed, CUBIC11—is the primary
cause of these problems. When bottle-
neck buffers are large, loss-based con-
gestion control keeps them full, causing
bufferbloat. When bottleneck buffers
are small, loss-based congestion con-
trol misinterprets loss as a signal of
congestion, leading to low throughput.
Fixing these problems requires an alter-
native to loss-based congestion control.
Finding this alternative requires an un-
derstanding of where and how network
Congestion and Bottlenecks
At any time, a (full-duplex) TCP connection has exactly one slowest link or
bottleneck in each direction. The bottleneck is
˲It determines the connection’s
maximum data-delivery rate. This is
a general property of incompressible
flow (for example, picture a six-lane
freeway at rush hour where an accident has reduced one short section to
a single lane. The traffic upstream of
the accident moves no faster than the
traffic through that lane).
˲ It is where persistent queues form.
Queues shrink only when a link’s departure rate exceeds its arrival rate. For
a connection running at maximum delivery rate, all links upstream of the bottleneck have a faster departure rate so
their queues migrate to the bottleneck.
Regardless of how many links a connection traverses or what their individual
speeds are, from TCP’s viewpoint an arbitrarily complex path behaves as a single link with the same RTT (round-trip
time) and bottleneck rate. Two physical
constraints, RTprop (round-trip propagation time) and BtlBw (bottleneck bandwidth), bound transport performance.
(If the network path were a physical pipe,
RTprop would be its length and BtlBw its
Figure 1 shows RTT and delivery
rate variation with the amount of data
in flight (data sent but not yet acknowledged). Blue lines show the RTprop
constraint, green lines the BtlBw constraint, and red lines the bottleneck
buffer. Operation in the shaded regions is not possible since it would vio-
Article development led by
Measuring bottleneck bandwidth
and round-trip propagation time.
BY NEAL CARDWELL, YUCHUNG CHENG, C. STEPHEN GUNN,
SOHEIL HASSAS YEGANEH, AND VAN JACOBSON