is a metric for the first bit, while latency is a metric for the entire data unit, which may contain more than one bit. In general:

latency = propagation delay + data unit size ÷ bandwidth

 

What this basically says is that the propagation delay is only part of the picture and that bandwidth affects performance as well. A look at the impact of the bandwidth in an example system shows why propagation delay is so important. Consider a 10-Gbps transmission system and a 1,250-byte (or equivalently, 10 Kbit, chosen both to reflect a reasonable maximum transmission unit with Ethernets and to make arithmetic easier!) data unit. The propagation time for the first bit in the NY/SF loop is 38 ms, and the last bit arrives a microsecond (10K/10G) later, making the total latency 38.001 ms. The majority of the latency is propagation delay. An interesting arithmetic exercise is to compute the distance at which a transmission system’s latency is double the propagation delay. For a 10-Gbps transmission system and 10-Kbit data-unit size, this is about 214 meters, or a few city blocks. For smaller data units or longer distances, propagation delay is the majority of the latency. (John Shaffer and I offer more detail on propagation delay versus latency in a previous paper.⁴)

It is instructive to take a few measurements to see what is what. Using the ping utility to send ICMP ECHO packets, I measured the round-trip latency between the University of Pennsylvania (klondike.cis.upenn.edu) and

FIGURE

1

Disparate Network Types

H1

H2

H3

Ethernet

H7

R3

H8

R1

point-to-point link (e.g., ISDN)

H4

FDDI ring

R2

H5

H6

Stanford University (cs.stanford.edu), two well-connected sites, as being about 87.5 ms. Rounding this to 88 ms and subtracting the fiber propagation time of 38 ms leaves a 50-ms difference. (Note that the New York–San Francisco numbers are assumed to be roughly equivalent to those from Philadelphia to Palo Alto.) Since these data units are only about 500 bits long, bandwidth between Penn and Stanford would have to be pretty bad (500 bits in 50 ms would be about 10 Kbps!) to explain the delay. So what could be the cause?

There are at least a couple of possible factors, both of which can be explained with the notion of hops. To understand hops, it helps to understand how a network differs from our 8,250-km loop of fiber. A real network is constructed of many interconnected pieces—for example, local area networks and wide area networks. Figure 1 represents a real physical network topology, with many types of networks and multiple devices. Hosts are labeled with H, routers with R, and network types are shown to be multiple in nature.

Many different packet formats and data units are in use, and the genius of the Internet is that it has a solution to make them all work together. This interoperability layer consists of a packet format and an address that are interpreted by devices called IP routers. The subnets interconnecting the routers can use whatever technology they choose as long as they can carry encapsulated IP packets between routers. Each router-router path is called a hop. As before with ping, it is instructive to obtain a measurement, which I obtained using traceroute between the two hosts I used before. There are 17 hops reported. Our analysis of an unobstructed fiber did not account for these routers, nor for the possibility that packets did not travel “as the crow flies” between the source and destination. The total propagation delay through this network, then, is equal to the sum of the propagation time across each subnet, plus the time required to pass through the routers.

Modern routers such as the Cisco CRS-1 exhibit average latencies of about 100 microseconds.² Our Philadelphia–Palo Alto example would include roughly 30 of them in the round-trip path, making the total router latency about 3 ms. The other causes of delay are more difficult to measure. Routers attempt to optimize a path between two points, but that may be difficult, so in addition to the delay through the routers we can expect a

Are Overcome

certain delay caused by path selections that deviate from

by Internetworking

a straight lines. Other possible sources of delay include

Technology

slower routers and other intervening appliances (such as firewalls), queuing (to wait for access to a busy shared