Like chocolate cake, you have to bake it before you eat it. Cognitive scientists understand this. When they use eye-tracking studies to learn how we process information, they actively take account of all relevant work, no matter the methodology or the discipline. When vendors promote eye tracking as easy and accessible, they gloss over that work, and because the heat maps look scientific, we fall for it.

It’s easy to understand why eye-tracking maps are so easily mistaken for findings. Humans intuit that data is messy, so if it looks nice, it must be analysis-ready. Unfortunately, because eye-tracking is so deceptively easy, it enables enormous fallacies in user research. It’s marvelous at proving other people wrong (“See, I told you green wouldn’t work”), proving our own points (“If the button were red, people would see it”), drawing shaky conclusions (“It’s not that people don’t want to use it, it’s that they don’t see it”) and discrediting our profession (“This isn’t so hard. Remind me again why we’re paying an expert to do this?”).

Like the other shortcuts I’ve
mentioned here, eye tracking gives
a dangerous amount of latitude
for anybody to make their own
guesses and draw their own con-
clusions. Eye-tracking data seems
very approachable, and it looks fun
to play with. However, its data is
stripped of all meaning and context,
and when we take it at face value, we
run the risk of drawing unsubstan-
tiated conclusions. Unfortunately,
our clients may also mistake eye-
tracking data for insights, and it’s
our responsibility to ensure they
don’t draw unsubstantiated conclu-
sions either. Our clients (who are not
trained in the fine art of considering
data in a holistic context) need solid
information to make solid business
decisions. In supporting that need,
we must ensure that our insights are
rich and that they provide informa-
tion our clients can trust.

will notice the problems. This seed grows into distrust of our individual work and has the potential to scatter seeds of distrust of the user-research profession.

Many shortcuts share the shiny allure of modern, sophisticated-looking technology, but in the end they are a poor substitute for our critical-thinking skills. They might look good, but they are compromises, and they don’t replace the fundamental skills we should be developing. These skills are not new; they are the skills polished by curious people across all scientific fields: Once we have made an observation and defined the problem, we form a hypothesis and test it. Those skills take a lifetime to perfect, and when we are neck-deep in fads, we can’t hone them. We might suffer the illusion that superficial understandings will suffice, and we might conclude that our restless minds are at their sharpest when wielding the new-est of an endless series of gadgets, but in reality we are letting the best things about us—our curiosity and our intellect—waste away.

This is perhaps the saddest thing about shortcuts. While we are leaping from gimmick to gimmick, we forget the reason we started poking around and asking questions and knitting our brows in puzzlement. We forget about the basic human need, as old as the wheel, to understand the world and its people. This is a huge undertaking. We should do it right.

What Can We Do, Then?

These are pitfalls for new and seasoned user researchers alike. Folks new to the field, including those transitioning from research in guerrilla-style environments, might inappropriately adapt techniques they already know, or they might address weak points in their research by Band-Aiding over them. Seasoned practitioners might tire of dealing with stakeholders who don’t care about deep, rich data, so they might look for ways to develop more insights faster, or yield to bad compromises.

Shortcuts, in all their varied and sneaky and tricky disguises, can entrap anybody along the entire spectrum of experience and cause our work to suffer. Even if our enthusiastic adoption of shiny things distracts us from noticing the weaknesses in our research, others

ABOUT THE AUTHOR
Leanna Gingras is the user
research coordinator at ITHAKA
and a problem solver by trade. As
part of her calling to create holis-
tic and delightful experiences, she
manages research studies, con-
ducts social experiments on teammates, and jug-
gles between quantitative and qualitative analysis.

March + April 2012

DOI: 10.1145/2090150.2090168
© 2012 ACM 1072-5220/12/03 $10.00