adaptive routing is 1.0. The results show that for most links, VLB performs about the same as the other two schemes. For the most heavily loaded link in each scheme, VLB’s link capacity usage is at worst 20% higher than that of the other two schemes. Thus, evaluations on actual data center workloads show that the simplicity and universality of VLB costs relatively little capacity when compared to much more complex traffic engineering schemes. Moreover, adaptive routing schemes might be difficult to implement in the data center. Since even the “elephant” flows are about 100MB (see Figure 2), lasting about 1s on a server with a 1Gbps NIC, any reactive traffic engineering scheme will need to run at least as frequently if it wants to react to individual flows.

Cost and Scale: With the range of low-cost commodity devices currently available, the VL2 topology can scale to create networks with no oversubscription between all the servers of even the largest data centers. For example, switches with 144 ports (D = 144) are available today for $150K, enabling a network that connects 100K servers using the topology in Figure 4 and up to 200K servers using a slight variation. Using switches with D = 24 ports (which are available today for $8K each), we can connect about 3K servers. Comparing the cost of a VL2 network for 35K servers with a conventional one found in one of our data centers shows that a VL2 network with no oversubscription can be built for the same cost as the current network that has 1:240 oversubscription. Building a conventional network with no oversubscription would cost roughly 14× the cost of a equivalent VL2 network with no oversubscription.

7. ReLateD WoRK

Data center network designs: There is great interest in building data center networks using commodity switches and a Clos topology. 2, 11, 20, 21 The designs differ in whether they provide layer- 2 semantics, their traffic engineering strategy, the maturity of their control planes, and their compatibility with existing switches. Other approaches use the servers themselves for switching data packets. 1, 12,

figure 11. cDf of normalized link utilizations for VLB, adaptive, and best oblivious routing schemes, showing that VLB (and best oblivious routing) come close to matching the link utilization performance of adaptive routing.

1

0.8

Percentile Rank

0.6

0.4

0.2

VLB Adaptive Best-Oblivious

0

0

0.2

0.4

Link Utilizations (Normalized)

0.6 0.8

1

1. 2

13 VL2 also leverages the programmability of servers; however, it uses servers only to control the way traffic is routed as switch ASICs forward packets at less cost in power and dollars per Mbps.

valiant load balancing: Valiant introduced VLB as a randomized scheme for communication among parallel processors interconnected in a hypercube topology. 6 Among its recent applications, VLB has been used inside the switching fabric of a packet switch. 3 VLB has also been proposed, with modifications and generalizations, 18, 23 for oblivious routing of variable traffic on the Internet under the hose traffic model. 8

Scalable routing: The Locator/ID Separation Protocol9 proposes “map-and-encap” as a key principle to achieve scalability and mobility in Internet routing. VL2’s control plane takes a similar approach (i.e., demand-driven host-information resolution and caching) but adapted to the data center environment and implemented on end hosts. SEAT TLE17 proposes a distributed host-information resolution system running on switches to enhance Ethernet’s scalability.

Commercial networks: Data Center Ethernet (DCE) 4 by Cisco and other switch manufacturers shares VL2’s goal of increasing network capacity through multipath. These industry efforts are primarily focused on consolidation of IP and storage area network (SAN) traffic, and there are few SANs in cloud-service data centers. Due to the requirement to support loss-less traffic, their switches need much bigger buffers (tens of MBs) than commodity Ethernet switches do (tens of KBs), hence driving their cost higher.

8. suMMaRy

VL2 is a new network architecture that puts an end to the need for oversubscription in the data center network, a result that would be prohibitively expensive with the existing architecture.

VL2 benefits the cloud service programmer. Today, programmers have to be aware of network bandwidth constraints and constrain server-to-server communications accordingly. VL2 instead provides programmers the simpler abstraction that all servers assigned to them are plugged into a single layer- 2 switch, with hot spot–free performance regardless of where the servers are actually connected in the topology. VL2 also benefits the data center operator as today’s bandwidth and control plane constraints fragment the server pool, leaving servers (which account for the lion’s share of data center cost) underutilized even while demand elsewhere in the data center is unmet. Instead, VL2 enables agility: any service can be assigned to any server, while the network maintains uniform high bandwidth and performance isolation between services.

VL2 is a simple design that can be realized today with available networking technologies, and without changes to switch control and data plane capabilities. The key enablers are an addition to the end-system networking stack, through well-established and public APIs, and a flat addressing scheme, supported by a directory service.

VL2 is efficient. Our working prototype, built using commodity switches, approaches in practice the high level