systems were available 15 years ago.
HPC systems rely on fast (low-latency)
and efficient interconnection networks capable of providing both high
bandwidth and efficient messaging for
fine-grained (for example, cache-line
size) communication. This zealous
attention to performance and low latency migrated to financial enterprise
systems where a fraction of a microsecond can make a difference in the
value of a transaction.
In recent years, Ethernet networks
have made significant inroads into
bridging the performance and scalability gap between capacity-oriented
clusters built using COTS (
commod-ity-off-the-shelf) components and
purpose-built custom system architectures. This is evident from the growth
of Ethernet as a cluster interconnect
on the Top500 list of most powerful
computers ( top500.org). A decade
ago high-performance networks were
mostly custom and proprietary interconnects, and Ethernet was used by
only 2% of the Top500 systems. Today,
however, more than 42% of the most
powerful computers are using Gigabit
Ethernet, according to the November
2011 list of Top500 computers. A close
second place is InfiniBand, which
is used by about 40% of the systems.
These standards-based interconnects
combined with economies of scale
provide the genetic material of a modern data-center network.
A modern data center,
13, 17, 24 as
shown in Figure 1, is home to tens of
thousands of hosts, each consisting of
one or more processors, memory, network interface, and local high-speed
I/O (disk or flash). Compute resources
are packaged into racks and allocated
as clusters consisting of thousands of
hosts that are tightly connected with
a high-bandwidth network. While the
network plays a central role in the overall system performance, it typically
represents only 10%–15% of the cluster cost. Be careful not to confuse cost
with value—the network is to a cluster
computer what the central nervous system is to the human body.
A Guided Tour
Article development led by
By Dennis ABTs AnD BoB feLDeRMAn
A good user experience depends on predictable
performance within the data-center network.
the MaGIC of the cloud is that it is always on and
always available from anywhere. Users have come to
expect that services are there when they need them.
A data center (or warehouse-scale computer) is the
nexus from which all the services flow. It is often
housed in a nondescript warehouse-sized building
bearing no indication of what lies inside. Amidst the
whirring fans and refrigerator-sized computer racks is
a tapestry of electrical cables and fiber optics weaving
everything together—the data-center network. This
article provides a “guided tour” through the principles
and central ideas surrounding the network at the heart
of a data center—the modern-day loom that weaves
the digital fabric of the Internet.
Large-scale parallel computers are grounded in HPC
(high-performance computing) where kilo-processor