out the goals of the study and what we need to learn, and then find ways to season those questions with analytics. If we’re concerned that a new form field will frustrate users, before we conduct a usability test we compare before-and-after numbers on form-completion rates, exit page destinations, and time on page. This helps identify things to watch for in the lab. If analytics show a pattern of folks heading en masse for the “about our company” link, we can keep a special eye on our in-person participants’ behavior and expectations around that link.

At the end of the day, our job is to solve problems. Analytics, like anything else, is a means to an end. Taking an iterative, systematic, and rigorous approach to problem solving yields a clearer connection between the problem, the research done around it, and the design that gets to the root of the issue.

March + April 2012

interactions

The Drive-by Analysis Shortcut:
Skimping on Thinking

From fear of analysis paralysis (spending too much time poring over data, needlessly turning over stones, and beating long-dead horses), we can swing to the other extreme and rush through analysis. I have learned that when something does need to be examined thoroughly, nothing can substitute for the grunt work of teasing out answers and squinting our eyes to see if the puzzle is yet complete.

Web analytics readily fall into this trap. They are indispensable and you will have to pry them from my cold dead fingers, but they are stunningly easily to screw up. Here is a basic example that many of us have grappled with: “Time on page is up 20 percent since last month!” If we take statistics like these at face value, we might consider it a win, but we need to dig deeper to figure out what the numbers mean. What kind of design changes have we made since then? Do people spend more time on that page because it takes longer to get stuff done? Did our South Korean users abandon us, taking with them their stunningly high-speed Internet? Increased time on page is just one of many deceptively simple numbers that, without context, raises more questions than it answers.

No single bit of analytics data can stand by itself, and analytics is at its most powerful when it’s part of a holistic picture of users, their goals, and their environment. When we bring together insights from other qualitative and quantitative studies, our understanding of the truth sharpens. Because of this, I approach analytics in much the same way that I approach salt: as an essential seasoning to (almost) every main dish.

For example, while planning an in-person usability test, I first sniff

The Square Peg in Round Hole Shortcut: Using the Wrong Tool for the Job After moving to a new apartment in an unfamiliar part of town, I drove to work using a familiar route that added five miles to my daily commute. I did this not just once or twice, but every workday for over a month. I vaguely knew there was another road to town, but I was afraid of getting lost, so I didn’t venture out of my comfort zone. In user research, it’s also tempting to stay inside our comfort zone and stick to tried-and-true methodologies even when they are not appropriate for the job. I’ve watched professional user researchers adapt them, stretch them, and hack them together with other methodologies. Inevitably, these “franken-methodologies” resemble a snake with legs stapled onto it, sadly attempting something for which it was never built.

We are not always as careful as we should be when planning research. Once we have identified a research need, it’s fantastically handy to have a wide range of tools and approaches to pick from to address that need. However, different tools answer our questions from different angles, and sometimes we simply pick the wrong angle, ending up with empty or inaccurate answers. All methodologies bias research in some way; when we understand what our bias is, we can take steps to address it. It’s therefore essential to understand the strengths and weaknesses of our tools, and what implications that holds for our results.

And here’s a good example: Eye tracking, in all of its popular glory, is a notoriously misapplied methodology. Eye-tracking technology monitors where and for how long people’s eyes fixate on a target. The original idea back in the day was to learn how people read and to correlate eye fixation with cognition. It was long the exclusive tool of labs with very deep pockets, but times have changed, and at UX conferences these days you can’t throw a rock without hitting an eye-tracking vendor. These vendors claim to deliver the power of the eye-tracking lab at dirt-cheap prices. Eye-tracking presentations and seminars (often given by said vendors) spring up like weeds, offering “eye-tracking 101” and “eye-tracking boot camp.” It’s not so expensive, they promise, and not so hard. Anybody can do it.

Great! What’s the catch? Well, eye tracking in UX is based on the premise that the resulting heat maps will reveal thoughts that users don’t verbalize, because they are not conscious of their attention processes. Unfortunately, the heat-map data does not actually represent the user’s mental processes.