figure 5. synchronous-write latency to a two-disk pool.
is mirrored across pairs of disks, and
writes to stable storage are not considered completed until the ZIL exists on
both sides of the mirror; thus, the NFS
I/O latency is from the slowest disk in
the pair. This has given the heat map
a bias toward the higher latencies. A
similar and greater effect was seen for
single- and double-parity RAID (their
heat-map screenshots are not included here).
To summarize what we know about
the icy lake: lines come from single
disks, and disk pairs cause increasing
and decreasing latency to occur. The
actual reason for the latency difference over time that seeds this pattern
has not been pinpointed; what causes
the rate of increase/decrease to change
(change in slope seen in Figure 5) is also
unknown; and, the higher latency line
seen in the single-disk pool (Figure 4)
is also not yet understood. Visualizing
latency in this way clearly poses more
questions than it provides answers.
figure 6. synchronous writes to a mirrored pool of disks.
as a TXG (transaction group). The ZIL
is expected to be written sequentially,
and so the heat map is also as expected (with the exception of the line at
the top). This will differ for a two-disk
striped pool, since ZFS will have a
ZIL on each disk and write to them in
round-robin fashion. This was tested,
and Figure 5 shows the resultant NFS
latency on a two-disk pool and the disk
I/O latency from each disk in the pool.
The reason for increasing and decreasing latency can now be theorized: as
the latency on one disk increases, the
other disk continues to turn and by
the time a request is issued has a corresponding smaller latency. The heat
map in Figure 2 is an extension of this,
with 22 disks instead of two.
The reason for the slope in the first
place has still not been pinpointed.
The disks are writing to a steadily in-
creasing offset, which is expected to be
placed in a sequential manner along
a disk track (it’s up to the disk what it
really does with it). If the starting point
of rotation were fixed, the rotational
latency to each write would steadily in-
crease as the disk turns farther to reach
the increasing offset (until a full revolu-
tion is reached). The starting point isn’t
fixed, however; rather it is the end point
of the previous I/O, which includes the
starting offset and I/O size. Since each
offset delta and I/O size is the same, the
rotational latency to the next I/O should
also be the same. As with the single-
disk pool analysis, the slope may actu-
ally be a result of client and network la-
tency while the disks continue to rotate.
The reason the slope changes is also
unknown, as seen in Figure 5 between
20:10:00 and 20:10: 45.
the Rainbow Pterodactyl
As with the icy lake, the rainbow pterodactyl is another simple workload that
has produced a surprisingly complex
pattern. This time disk I/O latency is
examined on a system with 48 disks
across two JBOD (just a bunch of disks)
enclosures. A local workload was executed to investigate I/O bus throughput by adding disks one by one with
sequential 128KB reads, while looking for knee points in the throughput graph. The latency was expected
to be consistent for the I/O size used
and appears as a narrow line, perhaps
with a slight increase as contention
increased on I/O subsystem buses
(which include Hyper Transport, PCIe,
and SAS). When one of those buses
reaches saturation, the latency is expected to increase much more sharply.
Therefore, only two features were expected: a gradual increase with consistent latency and later a sharp increase
with latency becoming less consistent
because of contention.
Figure 7 shows the throughput
graph and latency heat map from this
test. A new disk was added to the workload every two seconds, and a knee
point in the disk throughput plot can
be seen at 17: 55. Finding this knee
point was the original intent of this experiment; it was the latency heat map