We used these experiences to design Watchtower, our
third-generation cluster fabric. The key idea was to leverage the next-generation merchant silicon switch chips,
16x10G, to build a traditional switch chassis with a backplane.
Figure 6 shows the half rack Watchtower chassis along with
its internal topology and cabling. Watchtower consists of
eight line cards, each with three switch chips. Two chips
on each linecard have half their ports externally facing, for
a total of 16x10GE SFP+ ports. All three chips also connect
to a backplane for port to port connectivity. Watch-tower
deployment, with fiber cabling as seen in Figure 6 was substantially easier than the earlier Firehose deployments.
The higher bandwidth density of the switching silicon also
allowed us to build larger fabrics with more bandwidth to
individual servers, a necessity as servers were employing an
ever-increasing number of cores.
Saturn was the next iteration of our cluster architecture.
The principal goals were to respond to continued increases
in server bandwidth demands and to further increase
maximum cluster scale. Saturn was built from 24x10G
A major concern with FH1.1 in production was deploying
an unproven new network technology for our mission criti-
cal applications. To mitigate risk, we deployed Firehose 1. 1
in conjunction with our legacy networks as shown in Figure 5.
We maintained a simple configuration; the ToR would for-
ward default traffic to the legacy network (e.g., for connec-
tivity to external clusters/data centers) while more specific
intra-cluster traffic would use the uplinks to Firehose 1. 1.
We built a Big Red Button fail-safe to configure the ToRs to
avoid Firehose uplinks in case of catastrophic failure.
3. 2. Watchtower and Saturn: Global deployment
Our deployment experience with Firehose 1. 1 was largely
positive. We showed that services could enjoy substantially
more bandwidth than with traditional architectures, all
with lower cost per unit bandwidth. The main drawback to
Firehose 1. 1 was the deployment challenges with the external copper cabling.
Aggregation Block ( 32×10G to 32 spine blocks)
Stages 2, 3 or 4 linecard:
4×10G up, 4×10G down
Buddied ToR switches:
Each ToR has 2×10G up,
2×10G side, 48×1G down
Figure 4. Firehose 1. 1 packaging and topology. The top left picture
shows a linecard version of the board from Figure 3. The top right
picture shows a Firehose 1. 1 chassis housing six such linecards.
The bottom figure shows the aggregation block in Firehose 1. 1,
which was different from Firehose 1.0.
Legacy network routers
ToR ToR ToR
Firehose 1. 1 fabric
Figure 5. Firehose 1. 1 deployed as a bag-on-the-side Clos fabric.
32x10G to 32 aggregation blocks
Aggregation Block ( 32×10G to 32 spine blocks) Stages 2, 3 or 4 board
Stage 5 board:
ToR (Stage 1) board:
2×10G up, 24×1G down
Stages 2, 3 or 4 board:
4×10G up, 4×10G down
Figure 3. Firehose 1.0 topology. Top right shows a sample 8x10G port
fabric board in Firehose 1.0, which formed Stages 2, 3 or 4 of the
Figure 6. A 128x10G port Watchtower chassis (top left). The internal
non-blocking topology over eight linecards (bottom left). Four
chassis housed in two racks cabled with fiber (right).