Interactions - November/December 2014

INTERACTIONS.ACM.ORG 82 INTERACTIONS NOVEMBER–DECEMBER2014 FORUM EVALUATION AND USABILITY

is to interview a much larger sample that includes practitioners representing other countries and cultures to learn more about the role of usability testing and usability evaluation methods in UX elsewhere. To the extent that my provocative assertion that academics have too little exposure to usability practice outside universities to speak authoritatively about practice is correct, I strongly recommend that my academic colleagues begin to seek out such exposure. Given that relatively few of our graduates end up as academic researchers studying some aspect of HCI, it follows that most of them become practitioners. Some are likely to work in UX teams. One easy way to learn more about what they do would simply be to stay in touch once they graduate.

With respect to methods that practitioners use or could use, there seems to be plenty of room for novel evaluation methods, but it seems likely that empirical methods will remain a cornerstone of UX practice. Inspection methods might be useful, but only if practitioners are actually part of their development.

ACKNOWLEDGMENTS

I wholeheartedly thank the 12 interviewees who so freely gave me their time and information, typically at great inconvenience to them due to our different time zones.

Endnotes

1. Stewart, T. Usability (editorial). Behaviour

& Information Technology 28, 2 (2009), 99–100.

2. Dray, S. Engaged scholars, thoughtful practitioners: The interdependence of academics and practitioners in user-centered design and usability. Journal of Usability Studies 5, 1 (2009), 1–7.

3. McAllister, G. and White, G.R. Video game development and user experience.

In Evaluating the User Experience in Games. R. Bernhaupt, ed. Springer Verlag, Heidelberg, Germany, 2010, 107–130.

Gitte Lindgaard is professor of strategic design at Swinburne University of Technology in Melbourne, Australia, and Distinguished Research Professor (HCI) at Carleton University in Ottawa, Canada. As a human factors expert in telecommunications for many years, she has ample first-hand experience on both sides of the academic/practitioner fence.

the term heuristic evaluation, instead doing what they call an expert review or design review. The term heuristic, they said, is meaningless to people outside the HCI community, and the purpose of evaluation is to focus on the user interaction, rather than getting bogged down by particular heuristics.

The actual process of review and inspection was subtly similar, but not standard, from person to person or from team to team in most cases.

The variations in the process and rules guiding these inspections were almost endless. It was also clear that this sample of practitioners believed that the value of inspection methods was limited to the above-mentioned situations. Although research is afoot to improve heuristic evaluation methods, especially in the area of computer games, “feedback from developers suggests that they [heuristics] are too generic to be of much use” [ 3]. These practitioners would almost certainly agree.

USER-BASED
USABILITY EVALUATION

All 12 interviewees expressed a strong preference for, and almost religious adherence to, traditional user-based usability tests, even in organizations using agile software development processes, and even including computer game providers.

As one consultant said, “There is just no other or better way to do it [collect diagnostic usability data].” According to

10 interviewees, empirical user-based
tests made up 70 to 90 percent of their
work. It did not matter if these were
conducted in a dedicated usability lab
or in the field. Interviewees were all
acutely aware of the potential biases in
sampling methods and sample sizes,
limited experimental control in the
lab and in the field, and so on, but for
these participants, it would appear
that quasi-experimental methods are
the best we have and that they seem to
work.

The interviewees’ clear distinction in the relative value of opinion-based inspection methods and user-based empirical methods is a far cry from the often-blurred distinction between these methods discussed in the academic literature. Comparing these two method types is much like comparing apples with oranges; no matter their similarities, they remain distinctly different and their outcomes differ. In addition, when one’s job depends on delivering usability results and recommendations that help to fix problems, the detection of usability stumbling blocks is only half the equation. Problem detection is necessary and important but certainly not sufficient! Continued focus in the literature on detecting, rather than fixing, usability problems is therefore of neither theoretical nor practical value.

I was somewhat surprised to find that even with the current emphasis on UX rather than merely on usability, traditional empirical user-based evaluation was still as much in vogue today as it was back in my own former life as a human factors/ HCI practitioner. Then again, for an interactive experience to be engaging and fun, it must also be usable. It seems to me that the HCI community needs a selection of enhanced, augmented, or additional evaluation methods rather than replacements of empirical methods that have served us well since the 1970s.

NEXT STEPS

The backgrounds, skills, and job demands of the interviewees in this sample vary considerably; therefore, it is possibly unrepresentative of the larger population of practitioners. Four of them graduated from my Carleton University lab, so they were contaminated by our own worldviews. The fact that they all work in English-speaking countries is another potential bias. The fact that they work in vastly different kinds of organizations may be a strength or a weakness. My aim now