I
M
A
G
E
B
Y
X
C
A
G
E
S
resources like a bag of nails. Any single
nail doesn’t matter very much. It’s the
aggregate number of useful nails that
gets the house built. Typically, you buy
too many nails, expecting a certain fail-
ure rate. When a nail bends or breaks,
you don’t get too emotional about it.
Similarly, when a server in the data-
center breaks, you cannot get emotion-
al. It happens all the time.
Variety leads to complexity. Com-
plexity leads to unpredictability and a
lack of website availability.
The law of large numbers is prob-
ably true. The law of large numbers says
that individual pieces may be unreliable
but the aggregate expected failure rate
is predictable. While you cannot know
the exact outcome of the toss of a pair
of dice, you can know the expected rate
over time. The more tosses you have, the
better the confidence in the average.
Similarly, in a large datacenter, the
more servers you run, the easier it is to
plan for expected failures. The larger
the set of servers, the more likely some-
thing is broken. Figure 1 shows a typical
server failure rate.
It’s predictability that matters, not
reliability. Stuff can fail… Stuff will fail.
With tens of thousands of servers, lots
of stuff fails every day! You just need to
predict how often.
Hardware at Web Scale
Typical Web-scale datacenters think
about hardware differently from typical
IT shops. They avoid variety and stick to
common hardware as often as possible.
It’s assumed that each piece of hard-
ware may very well fail and the software
running on it needs to work success-
fully when that happens. Furthermore,
the inexorable pace of improvement
means it’s not cost effective to keep
hardware too long. The new stuff will
offer more “bang for the buck” and for
the electricity it consumes.
Bespoke is baroque. When some-
thing is bespoke, you tweak and modify
it until it is perfect. In some environ-
ments, systems and servers are designed
with their own special configuration.
Baroque is an architectural style with
a lot of ornate detail and tremendous
variety. In a large-scale environment,
bespoke becomes baroque. The individ-
