Interactions - November/December 2017

INTERACTIONS.ACM.ORG 82 INTERACTIONS NOVEMBER–DECEMBER2017 FORUM EVALUATION AND USABILITY

This forum addresses conceptual, methodological, and professional issues that arise in the UX field’s continuing effort to contribute robust information about users to product planning and design. — David Siegel and Susan Dray, Editors

Approximately seven months later, in
July 2016, the redesign was just about
ready to launch. To gather baseline
metrics on the current live site, we set up
a remote, unmoderated usability study
using Loop11. In the baseline study:
• There were 100 participants from
the four states in which this organization
primarily operates, recruited by a third
party.

• Each participant completed four tasks: 1) find a doctor, 2) find an urgent care facility, 3) find information about a specialty service (e.g., cancer care), and 4) determine how to pay a bill.

• There were four follow-up questions to help validate that the participants found the correct information.

• Participants completed a System Usability Scale (SUS) questionnaire.

• Participants answered a Net Promoter Score (NPS) question.

The baseline study results were disheartening (Figure 1). Overall task success was only 40 percent. While a “good” SUS is subjective, one published average is 68 [ 2], and the baseline SUS was 61. 2. A “good” NPS is also subjective. The scale ranges from -100 to 100, and published averages by industry range from 2 to 58 [ 3]. The NPS for this baseline study was - 7, far below any of those published averages.

While the team didn’t consider these figures desirable, it did make us hopeful that the comparison numbers we would collect after the launch of the redesign would show some improvement.

That comparison study was
conducted in December 2016,
approximately five months after the
redesign was launched. This delay
between the launch and the study
This case study concerns the
public-facing website of a multi-state
healthcare system headquartered in the
Midwestern U. S. Although the majority
of its more than 40,000 employees are
clinical practitioners and hospital and
practice administrators, the corporate
office is substantial and well staffed.

In terms of user experience (UX) maturity, this organization likely falls somewhere between “recognized” and “considered” on Earthy’s Human-Centeredness Scale [ 1], a relatively low maturity level. It typically employs one or two dedicated UX practitioners and periodically hires consultants to help in periods of heavy workload. They do iterate and test, but more in an ad-hoc way instead of as a clearly defined part of their digital product lifecycle.

In December 2015, the organization
hired me as a consultant to conduct a
lab-based, moderated, qualitative
usability study to ensure a planned
redesign was on the right track. This
study consisted of 11 one-hour, one-on-
one sessions in which participants were
asked to complete four tasks with each
of two clickable prototypes (one
representing the desktop experience
and one representing the mobile
experience) created in InVision.

The results of this qualitative study were mixed. Participants had positive reactions to the visual design of both the desktop and mobile home pages, but they often didn’t notice a key piece of information: which geographic area the site assumed they were in. Regarding the content of pages such as physician detail and facility detail, participants again had largely favorable opinions. For the screen showing the results of their searches (for a physician, hospital, or service), participants experienced a substantial amount of friction, but much of the difficulty was determined to be associated with confusing terminology and a few specific interaction design patterns in the filtering mechanism. Overall, we found that some parts of the prototype tested really well but that the issues of geographic area, filter presentation, and terminology should be subject to additional research.