1. Combined formative and summative evaluation.
In Halo 2 most of our user-testing efforts on the
single-player campaign were formative. Every ses-
sion was about drilling down into that week’s version
of the game, finding problems, and fixing problems
from the previous week. In Halo 3 we extended this
by adding a summative component to our measures.
It was important to us to track how we were doing on
several important high-level constructs from week
to week and over the course of all of our testing. We
were measuring the entire game experience as a
whole, as opposed to just focusing on the drill-down
2. Address broader organizational needs. We’re all
familiar with reasons why user experience profes-
sionals use real consumers as the major source of
data collection. What we need to remember, though,
is that many of our processes for data collection can
also be applied to those working on the product.
The Halo 3 team participated in internal playtesting
at strategic points in the development cycle to give
them a chance to experience the entire game experi-
ence in a “clean” way. This was critical, because it
forced the team to step out of their normal role and
become “player” again, giving them the ability to see
things they normally wouldn’t have.
Although this work has also been described in
some depth in Wired ( http://www.wired.com/wired/
issue/15-09/), in this essay we’ve presented the
theory behind the work and the strategic context of
Finally and most important, without the com-
mitment and creativity of the Bungie team, the best
research in the world would make no difference. Our
duty as user researchers is to produce the clearest,
timeliest, most holistic, and most actionable data
possible. We’ve outlined how we did that for Halo 3.
We wish every user research the opportunity to work
with a team like Bungie at least once in their career.
Life is short, have fun.
Permission to make digital
or hard copies of all or part
of this work for personal or
classroom use is granted
without the fee, provided
that copies are not made
or distributed for profit or
and that copies bear this
notice and the full citation
on the first page. To copy
otherwise, to republish,
to post on services or to
redistribute to lists, requires
prior specific permission
and/or a fee. © ACM
ABOUT THE AUTHORS Dennis Wixon leads a
team of more than 20 at Microsoft Game Studios,
which provides consulting and research to make
games fun. He is also a member of the User
Experience Leadership Team, a corporate steering
group. Dennis previously worked at Digital
Entertainment Corporation and has been an active member of CHI.
He has authored many articles on methodology and co-edited Field
Methods Casebook for Software Design (John Wiley & Sons).
Randy Pagulayan has led user research efforts on
multiple games at Microsoft Game Studios, includ-
ing Halo 2 (Xbox) and Halo 3 (Xbox 360). Randy
also co-authored several book chapters on user-
centered design in games and has been an invited
speaker for the Nielsen Norman User Experience
Event in 2003 and the Center for Computer Games Research at the
IT University of Copenhagen in 2005. Most recently, he was fea-
tured in a cover story in Wired magazine (September 2007).