INTERACTIONS.ACM.ORG 82 INTERACTIONS MARCH–APRIL2017
FORUM EVALUATION AND USABILITY
This forum addresses conceptual, methodological, and professional issues that arise
in the UX field’s continuing effort to contribute robust information about users to product
planning and design. — David Siegel and Susan Dray, Editors
Carol J. Smith, IBM
Getting the Most Out of
Remote Research and Testing
understand their vocalizations.
The limited or completely absent
facial/body language feedback can
potentially result in researchers missing
the full range of sarcasm, frustration,
distress, and boredom among other
“readable” emotions of participants.
It can therefore be difficult to perceive
when it is appropriate to ask the next
question. Even five seconds after asking
a question can seem like an eternity.
However, it’s also important to make
sure that the participants do not feel
abandoned. In some studies, I’ve
observed participants asking “Are you
still there?” when a facilitator is being
too quiet. Making audible listening
noises (“mmm”) that encourage them
to continue while avoiding words that
may be perceived as confirming can
help to facilitate the conversational tone
without intruding into their thoughts.
This is a difficult balance.
A pilot study is crucial for any usability
study; when the study is conducted
remotely, it is even more important.
Due to the lack of a moderator to
provide clarification, automated
remote studies require pilot studies to
ensure success. There is no safety net if
participants don’t understand the task.
They will simply fail, with no option
for either side to ask questions. Until
you watch the study, it is impossible to
know if it was successful. Additionally,
while some of the solutions remind
participants to think out loud, they are
typically more quiet than with in-person
or moderated studies.
To adequately prepare for an
automated study, I recommend
How do we manage the challenges
with remote sessions while
maintaining the level of quality
we need to answer our research
questions? This article describes
the benefits and primary challenges
with remote research and proposes
methods to support strong research.
Remote research and testing have
many benefits, both to participants
and to the study facilitators. The most
obvious benefits are logistical. The
participants do not need to go to a
lab, saving them time and effort, and
enabling them to maintain their normal
schedules. With automated (
nonmoderated) studies, they can participate
at their leisure, at the hours that are
most convenient to them.
As facilitators, we attain logistical
benefits through reduced travel and
less concern about providing correct
wayfinding to participants. In addition,
when observing the participants using
their own system, we can see how they
set up their desktop, how and when they
navigate between programs, and how
they use browser tabs (and sometimes
their Favorites). This insight into how
people work with their own machines
can be extremely helpful and is often
interesting as well.
For people with disabilities, remote
studies enable them to fully participate.
Those with visual disabilities can use
the systems they are most comfortable
with, such as screen readers, specialized
keyboards, larger screens, and
other personal devices. People with
disabilities that reduce their ease of
movement will not be frustrated by
having to arrange transit or concerns
about the study location’s accessibility.
The challenges and limitations of remote
studies are well established. It’s hard
enough for participants to focus when we
do in-person studies—focusing during a
remote study is even more challenging.
This is particularly true if the participant
has not been able to secure a private space.
Often participants are at their normal
desk, at work or in their home. As a result,
their normal interruptions are likely to
continue (urgent requests from their boss,
sleeping baby waking up, etc.).
In moderated and automated studies,
our work is “just” on a screen and will
be deprioritized against these other
situations. These interruptions can be
informative and helpful to research
but can also mean major disruption of
a usability study. In addition, there is
often background noise that can distract
participants and disrupt our ability to
For great remote research and testing:
→ Always run a pilot study.
→ Recruit loosely and minimally—
you can always run more studies.
→ Make a backup plan for