DRAM Errors in the Wild:
A Large-Scale Field Study
Errors in dynamic random access memory (DRAM) are a
common form of hardware failure in modern compute
clusters. Failures are costly both in terms of hardware replacement costs and service disruption. While a large body of
work exists on DRAM in laboratory conditions, little has been
reported on real DRAM failures in large production clusters.
In this paper, we analyze measurements of memory errors in a
large fleet of commodity servers over a period of 2. 5 years. The
collected data covers multiple vendors, DRAM capacities and
technologies, and comprises many millions of dual in-line
memory module (DIMM) days.
The goal of this paper is to answer questions such as the
following: How common are memory errors in practice? What
are their statistical properties? How are they affected by external factors, such as temperature and utilization, and by chip-specific factors, such as chip density, memory technology,
and DIMM age?
We find that DRAM error behavior in the field differs
in many key aspects from commonly held assumptions.
For example, we observe DRAM error rates that are orders
of magnitude higher than previously reported, with 25,000–
70,000 errors per billion device hours per Mb and more than
8% of DIMMs affected by errors per year. We provide strong
evidence that memory errors are dominated by hard errors,
rather than soft errors, which previous work suspects to be the
dominant error mode. We find that temperature, known to
strongly impact DIMM error rates in lab conditions, has a surprisingly small effect on error behavior in the field, when taking all other factors into account. Finally, unlike commonly
feared, we do not observe any indication that newer generations of DIMMs have worse error behavior.
Errors in dynamic random access memory (DRAM) devices
have been a concern for a long time. 3, 11, 15–17, 22 A memory error
is an event that leads to the logical state of one or multiple
bits being read differently from how they were last written.
Memory errors can be caused by electrical or magnetic interference (e.g., due to cosmic rays), can be due to problems with
the hardware (e.g., a bit being permanently damaged), or can
be the result of corruption along the data path between the
memories and the processing elements. Memory errors can
be classified into soft errors, which randomly corrupt bits but
do not leave physical damage; and hard errors, which corrupt
bits in a repeatable manner because of a physical defect.
The consequence of a memory error is system-dependent.
In systems using memory without support for error correction
and detection, a memory error can lead to a machine crash or
applications using corrupted data. Most memory systems in
server machines employ error correcting codes (ECC), 6 which
allow the detection and correction of one or multiple bit
errors. If an error is uncorrectable, i.e., the number of affected
bits exceed the limit of what the ECC can correct, typically a
machine shutdown is forced. In many production environments, including ours, a single uncorrectable error (UE) is
considered serious enough to replace the dual in-line memory
module (DIMM) that caused it.
Memory errors are costly in terms of the system failures
they cause and the repair costs associated with them. In
production sites running large-scale systems, memory
component replacements rank near the top of component
replacements19 and memory errors are one of the most
common hardware problems to lead to machine crashes. 18
There is also a fear that advancing densities in DRAM technology might lead to increased memory errors, exacerbating
this problem in the future. 3, 12, 13
Despite the practical relevance of DRAM errors, very little
is known about their prevalence in real production systems. Existing studies; for example, see Baumann, Borucki
et al., Johnston, May and Woods, Normand, and Ziegler
and Lanford, 3, 4, 9, 11, 16, 22 are mostly based on lab experiments using accelerated testing, where DRAM is exposed to
extreme conditions (such as high temperature) to artificially
induce errors. It is not clear how such results carry over
to real production systems. The few prior studies that are
based on measurements in real systems are small in scale,
such as recent work by Li et al., 10 who report on DRAM errors
in 300 machines over a period of 3–7 months. Moreover,
existing work is not always conclusive in their results. Li et
al. cite error rates in the 200–5000 FIT per Mb range from
previous lab studies, and themselves found error rates of < 1
FIT per Mb.
This paper provides the first large-scale study of DRAM
memory errors in the field. It is based on data collected from
Google’s server fleet over a period of more than 2 years making
up many millions of DIMM days. The DRAM in our study
covers multiple vendors, DRAM densities and technologies
(DDR1, DDR2, and FBDIMM).
The goal of this paper is to answer the following questions:
How common are memory errors in practice? How are they
affected by external factors, such as temperature, and system
utilization? How do they vary with chip-specific factors, such
as chip density, memory technology, and DIMM age? What
are their statistical properties?
The original version of this paper was published in
Proceedings of ACM SIGMETRICS, June 2009.