Sender-side Buffers and the Case
for Multimedia Adaptation
Aiman Erbad and Charles “Buck” Krasic
You Don’t Know Jack
about Network Performance
Kevin Fall and Steve McCanne
A Guided Tour through
Dennis Abts and Bob Felderman
1. Abrahamsson, M. TCP ACK suppression. IE TF AQM
mailing list, 2015; https://www.ietf.org/mail-archive/
2. Brakmo, L.S., Peterson, L.L. TCP Vegas: End-to-end
congestion avoidance on a global Internet. IEEE J.
Selected Areas in Communications 13, 8 (1995), 1465–1480.
3. Chakravorty, R., Cartwright, J., Pratt, I. Practical
experience with TCP over GPRS. IEEE GLOBECOM, 2002.
4. Corbet, J. TSO sizing and the FQ scheduler. LWN.net,
5. Ericsson. Ericsson Mobility Report (June), 2015;
6. ESnet. Application tuning to optimize international
astronomy workflow from NERSC to LFI-DPC at
7. Flach, T. et al. An Internet-wide analysis of traffic
policing. SIGCOMM, 2016, 468–482.
8. Gail, R., Kleinrock, L. An invariant property of
computer network power. In Proceedings of the
International Conference on Communications 63, 1
(1981), 1-63. 1. 5.
9. Gettys, J., Nichols, K. Bufferbloat: Dark buffers in the
Internet. acmqueue 9, 11 (2011); http://queue.acm.
10. Ha, S., Rhee, I. 2011. Taming the elephants: new TCP
slow start. Computer Networks 55, 9 (2011), 2092–2110.
11. Ha, S., Rhee, I., Xu, L. CUBIC: a new TCP-friendly
high-speed TCP variant. ACM SIGOPS Operating
Systems Review 42, 5 (2008), 64–74.
12. Heidergott, B., Olsder, G. J., Van Der Woude, J. Max
Plus at Work: Modeling and Analysis of Synchronized
Systems: a Course on Max-Plus Algebra and its
Applications. Princeton University Press, 2014.
13. Jacobson, V. Congestion avoidance and control. ACM
SIGCOMM Computer Communication Review 18, 4
14. Jaffe, J. Flow control power is nondecentralizable.
IEEE Transactions on Communications 29, 9 (1981),
15. Jain, S. et al. B4: experience with a globally-deployed
software defined WAN. ACM SIGCOMM Computer
Communication Review 43, 4 (2013), 3–14.
16. Kleinrock, L. 1979. Power and deterministic rules
of thumb for probabilistic problems in computer
communications. In Proceedings of the International
Conference on Communications (1979), 43. 1. 1-43. 1. 10.
17. Mathis, M., Semke, J., Mahdavi, J., Ott, T. The
macroscopic behavior of the TCP congestion
avoidance algorithm. ACM SIGCOMM Computer
Communication Review 27, 3 (1997), 67–82.
18. Wikipedia. GPRS core network serving GPRS support
Neal Cardwell, Yuchung Cheng, C. Stephen Gunn,
Soheil Hassas Yeganeh, and Van Jacobson are members
of Google’s make-tcp-fast project, whose goal is to evolve
Internet transport via fundamental research and open
source software. Project contributions include TFO (TCP
Fast Open), TLP ( Tail Loss Probe), RACK loss recovery,
fq/pacing, and a large fraction of the git commits to the
Linux kernel TCP code for the past five years. They can be
contacted at https://googlegroups.com/d/forum/bbr-dev.
Copyright held by owners/authors.
Cellular systems adapt per-subscrib-er bandwidth based partly on a demand estimate that uses the queue of
packets destined for the subscriber.
Early versions of BBR were tuned to
create very small queues, resulting
in connections getting stuck at low
rates. Raising the peak ProbeBW
pacing_gain to create bigger queues
resulted in fewer stuck connections,
indicating it is possible to be too nice
to some networks. With the current
1. 25 × BtlBw peak gain, no degradation is apparent compared with CUBIC on any network.
Delayed and stretched aks.
Cellular, Wi-Fi, and cable broadband
networks often delay and aggregate
ACKs. 1 When inflight is limited to
one BDP, this results in throughput-reducing stalls. Raising ProbeBW’s
cwnd_gain to two allowed BBR to
continue sending smoothly at the estimated delivery rate, even when ACKs
are delayed by up to one RTT. This
largely avoids stalls.
Token-bucket policers. BBR’s initial YouTube deployment revealed
that most of the world’s ISPs mangle
traffic with token-bucket policers. 7
The bucket is typically full at connection startup so BBR learns the underlying network’s BtlBw, but once
the bucket empties, all packets sent
faster than the (much lower than
BtlBw) bucket fill rate are dropped.
BBR eventually learns this new delivery rate, but the ProbeBW gain cycle
results in continuous moderate losses. To minimize the upstream bandwidth waste and application latency
increase from these losses, we added
policer detection and an explicit policer model to BBR. We are also actively researching better ways to mitigate the policer damage.
Competition with loss-based con-
gestion control. BBR converges to-
ward a fair share of the bottleneck
bandwidth whether competing with
other BBR flows or with loss-based
congestion control. Even as loss-
based congestion control fills the
available buffer, ProbeBW still ro-
bustly moves the BtlBw estimate
toward the flow’s fair share, and
ProbeRTT finds an RTProp estimate
just high enough for tit-for-tat con-
vergence to a fair share. Unmanaged
router buffers exceeding several
BDPs, however, cause long-lived loss-
based competitors to bloat the queue
and grab more than their fair share.
Mitigating this is another area of ac-
Rethinking congestion control pays
big dividends. Rather than using
events such as loss or buffer occupancy, which are only weakly correlated
with congestion, BBR starts from
Kleinrock’s formal model of congestion and its associated optimal operating point. A pesky “impossibility”
result that the crucial parameters
of delay and bandwidth cannot be
determined simultaneously is side-stepped by observing they can be estimated sequentially. Recent advances
in control and estimation theory are
then used to create a simple distributed control loop that verges on the
optimum, fully utilizing the network
while maintaining a small queue.
Google’s BBR implementation is
available in the open source Linux
BBR is deployed on Google’s B4
backbone, improving throughput by
orders of magnitude compared with
CUBIC. It is also being deployed on
Google and You Tube Web servers, substantially reducing latency on all five
continents tested to date, most dramatically in developing regions. BBR
runs purely on the sender and does
not require changes to the protocol,
receiver, or network, making it incrementally deployable. It depends only
on RTT and packet-delivery acknowledgment, so can be implemented for
most Internet transport protocols.
Thanks to Len Kleinrock for pointing
out the right way to do congestion control and Larry Brakmo for pioneering
work on Vegas2 and New Vegas congestion control that presaged many
elements of BBR, and for advice and
guidance during BBR’s early development. We also thank Eric Dumazet,
Nandita Dukkipati, Jana Iyengar, Ian
Swett, M. Fitz Nowlan, David Wether-all, Leonidas Kontothanassis, Amin
Vahdat, and the Google BwE and YouTube infrastructure teams.