message lengths (see Figure 3c); the
x-axis is labeled with message size but
is linear in number of messages, with
each of the 10 tickmarks corresponding to 10% of all messages. With this
view of the data it became easy to see
that the parameter matters, as it affected approximately 70% of all messages
in the experiment (those smaller than
approximately 5 Kbytes).
Figure 3c includes more information than the other graphs; in addition to displaying slowdown, it also
displays the CDF of message sizes via
the x-axis labels. As a result it is easy to
see that messages in this workload are
mostly short; 60% of all messages require no more than 960 bytes. Figure
3c makes it clear that Figure 3a and
Figure 3b are misleading.
The keys to good performance evaluation are a keen eye for things that do
not make sense and a willingness to
measure from many different angles.
This takes more time than the quick and
shallow measurements that are common
today but provides a deeper and more accurate understanding of the system being measured. In addition, if you apply
the scientific method, making and testing hypotheses, you will improve your
intuition about systems. This will result
in both better designs and better performance measurements in the future.
This article benefited from comments
and suggestions from Jonathan Elli-thorpe, Collin Lee, Yilong Li, Behnam
Montazeri, Seojin Park, Henry Qin,
Stephen Yang, and the anonymous
1. Nickerson, R.S. Confirmation bias: A ubiquitous
phenomenon in many guises. Review of General
Psychology 2, 2 (June 1998), 175–220.
2. Ousterhout, J. A Philosophy of Software Design.
Yaknyam Press, Palo Alto, CA, 2018.
3. Ousterhout, J., Gopalan, A., Gupta, A., Kejriwal, A.,
Lee, C., Montazeri, B., Ongaro, D., Park, S.J., Qin,
H., Rosenblum, M. et al. The RAMCloud storage
system. ACM Transactions on Computer Systems
33, 3 (Aug. 2015), 7.
4. Rosenblum, M. and Ousterhout, J.K. The design
and implementation of a log-structured file system.
ACM Transactions on Computer Systems 10, 1 (Feb.
John Ousterhout ( email@example.com) is the Bosack
Lerner Professor of Computer Science at Stanford
University, Stanford, CA, USA.
© 2018 ACM 0001-0782/18/7 $15.00
Figure 3. Each figure displays 99th-percentile slowdown (delivery time for messages
of a given size, divided by the best possible time for messages of that size) as a function
of message size in a given workload: (a) x-axis is linear; (b) x-axis is logarithmic; and
(c) x-axis is scaled to match the CDF of message lengths. Different curves correspond
to different settings of the “unscheduled bytes” parameter.
Message Size (Megabytes)
1 500 1000 9328 (RTTbytes) 18656 ( 2 x RTTbytes)
1e+02 4e+02 1e+03 4e+03 1e+04 4e+04 1e+05 3e+05 1e+06 3e+06 1e+07
Message Size (Bytes)
313 371 491 561 646 960 4582 486091203731e+07
Message Size (Bytes)