Figure 5. Crashes/day for a firmware bug. Patch was released via
Wu on day 10.
Reports as of peak
7 10 13 16 19 22 25 28
deployments. While WER does not make debugging in the
small significantly easier (other than perhaps providing programmers with better analysis of core dumps), WER has
enabled a new class of debugging in the large. The statistics
collected by WER help us to prioritize valued programmer
resources, understand error trends, and find correlated errors.
WER’s progressive data collection strategy means that
programmers get the data they need to debug issues, in
the large and in the small, while minimizing the cost of
data collection to users. Automated error analysis ensures
programmers are not distracted with previously diagnosed
errors. It also ensures that users are made aware of fixes
that can immediately improve their computing experience.
As applied to WER, the law of large numbers says that we
will eventually collect sufficient data to diagnose even rare
Heisenbugs3; WER has already helped identify such bugs
dating back to the original Windows NT kernel.
WER is the first system to provide users with an end-to-end solution for reporting and recovering from errors. WER
provides programmers with real-time data about errors
actually experienced by users and provides them with an
incomparable billion-computer feedback loop to improve
1. Bush, W.r., Pincus, J. D., sielaff, D.J.
a static analyzer for finding dynamic
programming errors. Softw. Pract. Exp.
30 ( 5) (2000), 775–802.
2. everett, r.r. the Whirlwind I computer.
In Proceedings of the 1951 Joint
AIEE–IRE Computer Conference
(Philadelphia, Pa), 1951.
3. gray, J. Why do computers stop and what
can we do about it. In Proceedings of the
6th International Conference on Reliability
and Distributed Databases, 1986, 3–12.
Kinshuman Kinshumann, Kirk Glerum,
Steve Greenberg, Gabriel Aul, Vince
Orgovan, Greg nichols, David Grant,
Gretchen Loihle, and Galen hunt
4. lee, I., Iyer, r.k. faults, symptoms,
and software fault tolerance in the
tandem guarDIan90 operating
system. In Digest of Papers of
the Twenty- Third International
Symposium on Fault-Tolerant
Computing (FTCS- 23). Ieee,
toulouse, france, 1993.
5. Walter, e.s., Wallace, V.l. further
analysis of a computing center
environment. Commun. ACM 10 ( 5)