Communications of the ACM

provided here, but interested readers can refer to http://hdrhistogram.org/6 for more details and a memory-efficient implementation.

From the previous description it should be apparent that HDR histograms span an extremely large range of values. Bin sizes are similar to float-number precisions: the larger the value, the less precision is available. In addition, bin boundaries are independent of the dataset. Hence, the aggregation technique described earlier applies to HDR histograms.

Histograms as heat maps. Observing the change of data distributions over time requires an additional dimension on the histogram plot. A convenient method of doing so is to represent the sample densities as a heat map instead of a bar chart. Figure 7 shows the request-rate data visualized in such a way. Light colors mean low sample density, dark colors signal high sample density.

Multiple histogram heat maps that were captured over time can be combined into a single two-dimensional heat map.

Figure 8 shows a particularly interesting example of such a visualization for a sequence of HDR histograms of Web-request latencies. Note that the distribution of the data is bimodal, with one mode constant around ~5ms and another more diffuse mode ascending from ~10ms to ~50ms. In this particular case the second mode was caused by a bug in a session handler, which kept adding new entries to a binary tree. This tree had to be traversed for each incoming request, causing extended delays. Even the logarithmic growth of the average traversal time can be spotted if you look carefully.

Classical Summary Statistics

The aim of summary statistics is to provide a summary of the essential features of a dataset. It is the numeric equivalent of an “elevator pitch” in a business context: just the essential information without all the details.

Good summary statistics should
be able to answer questions such as
“What are typical values?” or “How
much variation is in the data?” A de-
sirable property is robustness against
outliers. A single faulty measurement
should not change a rough description
of the dataset.
This section looks at the classical
summary statistics: mean values and
standard deviations. In the next sec-
tion we will meet their quantile-based
counterparts, which are more robust
against outliers.

Mean value or average of a dataset X = [x1,..., xn] is defined as

μ (x1,…,xn) = 1 n ∑ni= 1 xi

or when expressed as Python code:

def mean(X): return sum(X) / len(X)

The mean value has the physical interpretation of the center of mass if weights of equal weight are placed on the points xi on a (mass-less) axis.

When the values of xi are close together,
Mean values are abundant in IT op-
erations. One common application of
mean values is data rollup. When mul-
tiple samples arrive during a sampling
period of one minute, for example, the
mean value is calculated as a “one-
minute rollup” and stored instead of
the original samples. Similarly, if data
is available for every minute, but you
are interested only in hour intervals,
you can “roll up the data by the hour”
by taking mean values.