provided here, but interested readers
can refer to http://hdrhistogram.org/6
for more details and a memory-efficient implementation.
From the previous description it
should be apparent that HDR histograms span an extremely large range
of values. Bin sizes are similar to float-number precisions: the larger the
value, the less precision is available.
In addition, bin boundaries are independent of the dataset. Hence, the aggregation technique described earlier
applies to HDR histograms.
Histograms as heat maps.
Observing the change of data distributions
over time requires an additional dimension on the histogram plot. A convenient method of doing so is to represent the sample densities as a heat
map instead of a bar chart. Figure 7
shows the request-rate data visualized
in such a way. Light colors mean low
sample density, dark colors signal high
sample density.
Multiple histogram heat maps that
were captured over time can be combined into a single two-dimensional
heat map.
Figure 8 shows a particularly interesting example of such a visualization for a sequence of HDR histograms of Web-request latencies.
Note that the distribution of the data
is bimodal, with one mode constant
around ~5ms and another more diffuse mode ascending from ~10ms
to ~50ms. In this particular case the
second mode was caused by a bug in
a session handler, which kept adding
new entries to a binary tree. This tree
had to be traversed for each incoming request, causing extended delays.
Even the logarithmic growth of the
average traversal time can be spotted
if you look carefully.
Classical Summary Statistics
The aim of summary statistics is to
provide a summary of the essential
features of a dataset. It is the numeric
equivalent of an “elevator pitch” in a
business context: just the essential information without all the details.
Good summary statistics should
be able to answer questions such as
“What are typical values?” or “How
much variation is in the data?” A de-
sirable property is robustness against
outliers. A single faulty measurement
should not change a rough description
of the dataset.
This section looks at the classical
summary statistics: mean values and
standard deviations. In the next sec-
tion we will meet their quantile-based
counterparts, which are more robust
against outliers.
Mean value or average of a dataset
X = [x1,..., xn] is defined as
μ (x1,…,xn) = 1 n ∑ni= 1 xi
or when expressed as Python code:
def mean(X): return sum(X) / len(X)
The mean value has the physical
interpretation of the center of mass
if weights of equal weight are placed
on the points xi on a (mass-less) axis.
When the values of xi are close together,
Mean values are abundant in IT op-
erations. One common application of
mean values is data rollup. When mul-
tiple samples arrive during a sampling
period of one minute, for example, the
mean value is calculated as a “one-
minute rollup” and stored instead of
the original samples. Similarly, if data
is available for every minute, but you
are interested only in hour intervals,
you can “roll up the data by the hour”
by taking mean values.
Despite their abundance, mean
values lead to a variety of problems
when measuring performance of services. To quote Dogan Ugurlu from
Figure 10. Ping latency spike on a view range of 6H vs. 48H.
0.09
0.08
0.07
0.06
0.05
0.04
0.03
0.02
0.01
0.09
0.08
0.07
0.06
0.05
0.04
0.03
0.02
0.01
Nov 21
0:00
Nov 21
6:00
Nov 21
4:00
Nov 21
2:00
ms
m
s
Nov 20
12:00
Nov 22
0:00
Nov 21
12:00
Nov 22
12:00
Nov 21
0:00
6h view range:
all samples
are visible
48h view range
shows mean-value
aggregates
Figure 11. A request latency dataset.
standard deviation max deviation
mean value
0 0.02 0.04
Latency in ms
0.06 0.08
mean absolute deviation