INTERACTIONS.ACM.ORG 82 INTERACTIONS NOVEMBER–DECEMBER2014
FORUM EVALUATION AND USABILITY
is to interview a much larger sample
that includes practitioners representing
other countries and cultures to learn
more about the role of usability testing
and usability evaluation methods in
UX elsewhere. To the extent that my
provocative assertion that academics
have too little exposure to usability
practice outside universities to speak
authoritatively about practice is
correct, I strongly recommend that
my academic colleagues begin to
seek out such exposure. Given that
relatively few of our graduates end up
as academic researchers studying some
aspect of HCI, it follows that most of
them become practitioners. Some are
likely to work in UX teams. One easy
way to learn more about what they do
would simply be to stay in touch once
they graduate.
With respect to methods that
practitioners use or could use, there
seems to be plenty of room for novel
evaluation methods, but it seems likely
that empirical methods will remain a
cornerstone of UX practice. Inspection
methods might be useful, but only if
practitioners are actually part of their
development.
ACKNOWLEDGMENTS
I wholeheartedly thank the 12
interviewees who so freely gave me
their time and information, typically at
great inconvenience to them due to our
different time zones.
Endnotes
1. Stewart, T. Usability (editorial). Behaviour
& Information Technology 28, 2 (2009),
99–100.
2. Dray, S. Engaged scholars, thoughtful
practitioners: The interdependence of
academics and practitioners in user-centered design and usability. Journal of
Usability Studies 5, 1 (2009), 1–7.
3. McAllister, G. and White, G.R. Video
game development and user experience.
In Evaluating the User Experience in
Games. R. Bernhaupt, ed. Springer Verlag,
Heidelberg, Germany, 2010, 107–130.
Gitte Lindgaard is professor of strategic
design at Swinburne University of Technology
in Melbourne, Australia, and Distinguished
Research Professor (HCI) at Carleton
University in Ottawa, Canada. As a human
factors expert in telecommunications for many
years, she has ample first-hand experience on
both sides of the academic/practitioner fence.
the term heuristic evaluation, instead
doing what they call an expert review
or design review. The term heuristic,
they said, is meaningless to people
outside the HCI community, and the
purpose of evaluation is to focus on the
user interaction, rather than getting
bogged down by particular heuristics.
The actual process of review and
inspection was subtly similar, but
not standard, from person to person
or from team to team in most cases.
The variations in the process and
rules guiding these inspections were
almost endless. It was also clear
that this sample of practitioners
believed that the value of inspection
methods was limited to the above-mentioned situations. Although
research is afoot to improve heuristic
evaluation methods, especially in the
area of computer games, “feedback
from developers suggests that they
[heuristics] are too generic to be of
much use” [ 3]. These practitioners
would almost certainly agree.
USER-BASED
USABILITY EVALUATION
All 12 interviewees expressed a
strong preference for, and almost
religious adherence to, traditional
user-based usability tests, even in
organizations using agile software
development processes, and even
including computer game providers.
As one consultant said, “There is just
no other or better way to do it [collect
diagnostic usability data].” According to
10 interviewees, empirical user-based
tests made up 70 to 90 percent of their
work. It did not matter if these were
conducted in a dedicated usability lab
or in the field. Interviewees were all
acutely aware of the potential biases in
sampling methods and sample sizes,
limited experimental control in the
lab and in the field, and so on, but for
these participants, it would appear
that quasi-experimental methods are
the best we have and that they seem to
work.
The interviewees’ clear distinction
in the relative value of opinion-based inspection methods and user-based empirical methods is a far cry
from the often-blurred distinction
between these methods discussed in
the academic literature. Comparing
these two method types is much like
comparing apples with oranges; no
matter their similarities, they remain
distinctly different and their outcomes
differ. In addition, when one’s job
depends on delivering usability
results and recommendations that
help to fix problems, the detection of
usability stumbling blocks is only half
the equation. Problem detection is
necessary and important but certainly
not sufficient! Continued focus in the
literature on detecting, rather than
fixing, usability problems is therefore
of neither theoretical nor practical
value.
I was somewhat surprised to find
that even with the current emphasis
on UX rather than merely on
usability, traditional empirical user-based evaluation was still as much
in vogue today as it was back in my
own former life as a human factors/
HCI practitioner. Then again, for an
interactive experience to be engaging
and fun, it must also be usable. It seems
to me that the HCI community needs
a selection of enhanced, augmented,
or additional evaluation methods
rather than replacements of empirical
methods that have served us well since
the 1970s.
NEXT STEPS
The backgrounds, skills, and job
demands of the interviewees in this
sample vary considerably; therefore,
it is possibly unrepresentative of the
larger population of practitioners.
Four of them graduated from my
Carleton University lab, so they were
contaminated by our own worldviews.
The fact that they all work in English-speaking countries is another potential
bias. The fact that they work in vastly
different kinds of organizations may be
a strength or a weakness. My aim now
Most interviewees
said they have
abandoned the term
heuristic evaluation,
instead doing what they
call an expert review
or design review.