figure 5. synchronous-write latency to a two-disk pool.

is mirrored across pairs of disks, and writes to stable storage are not considered completed until the ZIL exists on both sides of the mirror; thus, the NFS I/O latency is from the slowest disk in the pair. This has given the heat map a bias toward the higher latencies. A similar and greater effect was seen for single- and double-parity RAID (their heat-map screenshots are not included here).

To summarize what we know about the icy lake: lines come from single disks, and disk pairs cause increasing and decreasing latency to occur. The actual reason for the latency difference over time that seeds this pattern has not been pinpointed; what causes the rate of increase/decrease to change (change in slope seen in Figure 5) is also unknown; and, the higher latency line seen in the single-disk pool (Figure 4) is also not yet understood. Visualizing latency in this way clearly poses more questions than it provides answers.

 

figure 6. synchronous writes to a mirrored pool of disks.

 

as a TXG (transaction group). The ZIL is expected to be written sequentially, and so the heat map is also as expected (with the exception of the line at the top). This will differ for a two-disk striped pool, since ZFS will have a ZIL on each disk and write to them in round-robin fashion. This was tested, and Figure 5 shows the resultant NFS latency on a two-disk pool and the disk I/O latency from each disk in the pool. The reason for increasing and decreasing latency can now be theorized: as the latency on one disk increases, the other disk continues to turn and by the time a request is issued has a corresponding smaller latency. The heat map in Figure 2 is an extension of this, with 22 disks instead of two.

The reason for the slope in the first
place has still not been pinpointed.
The disks are writing to a steadily in-
creasing offset, which is expected to be
placed in a sequential manner along
a disk track (it’s up to the disk what it
really does with it). If the starting point
of rotation were fixed, the rotational
latency to each write would steadily in-
crease as the disk turns farther to reach
the increasing offset (until a full revolu-
tion is reached). The starting point isn’t
fixed, however; rather it is the end point
of the previous I/O, which includes the
starting offset and I/O size. Since each
offset delta and I/O size is the same, the
rotational latency to the next I/O should
also be the same. As with the single-
disk pool analysis, the slope may actu-
ally be a result of client and network la-
tency while the disks continue to rotate.
The reason the slope changes is also
unknown, as seen in Figure 5 between
20:10:00 and 20:10: 45.

the Rainbow Pterodactyl

As with the icy lake, the rainbow pterodactyl is another simple workload that has produced a surprisingly complex pattern. This time disk I/O latency is examined on a system with 48 disks across two JBOD (just a bunch of disks) enclosures. A local workload was executed to investigate I/O bus throughput by adding disks one by one with sequential 128KB reads, while looking for knee points in the throughput graph. The latency was expected to be consistent for the I/O size used and appears as a narrow line, perhaps with a slight increase as contention increased on I/O subsystem buses (which include Hyper Transport, PCIe, and SAS). When one of those buses reaches saturation, the latency is expected to increase much more sharply. Therefore, only two features were expected: a gradual increase with consistent latency and later a sharp increase with latency becoming less consistent because of contention.

Figure 7 shows the throughput graph and latency heat map from this test. A new disk was added to the workload every two seconds, and a knee point in the disk throughput plot can be seen at 17: 55. Finding this knee point was the original intent of this experiment; it was the latency heat map