FORUM EVALUATION AND USABILITY
for usability studies, what you are
testing may be difficult to use or
unstable. In those situations, having
a technical person on hand to assist is
Assisting participants is always
a challenge, as their system setup is
typically unique to them, making it
difficult to troubleshoot problems.
When screen-sharing software is used,
we can attempt to troubleshoot by
walking them through various tasks
and have a chance of salvaging the
sessions. However, in some cases they
may not be able to activate the screen
share, or there is not one to use. In those
situations, a tool like Copilot (https://
www.copilot.com/) can enable a direct
line into their computer to diagnose and
address the issues.
Always have a backup plan to use
another type of software. Use screen-sharing software that enables you
to give participants control in case
they cannot (or will not) share their
screen. Skype and/or similar social
media/screen-sharing solutions can
be used as a last-minute backup. The
key is to make sure it is something
you are familiar with. For moderated
studies, have a note taker ready to take
screenshots if you are unable to record
within the software.
Finally, please keep in mind that
mistakes will be made. People will miss
their sessions and technology will fail.
Empathize with your participants,
apologize, and try to salvage the session
and/or reschedule when appropriate.
Special thanks to the UX Designers
Pittsburgh MeetUp May 2016 workshop
participants, who informed this work
through their participation.
1. Krug, S. Don’t Make Me Think! A Common
Sense Approach to Web Usability (First ed.).
New Riders. Berkeley, CA, 2000.
Carol Smith conducted her first remote
usability studies in 2004. She has a master’s
degree in human-computer interaction
from DePaul University and is an active UX
community organizer. She is currently senior
design manager for IBM Watson and lives in
situation. There will be cases when
you will need to recruit more tightly
to match a particular type of user,
and what may be considered loose
recruiting to one organization may be
overly stringent to another.
With some solutions, you can
provide a list of potential participants
to the company hosting the study.
In other situations, you can invite
people to the study yourself. The key
is to ensure that participants have
the technology and the ability to
participate. Their ability to participate
includes both the ability to install
the software and the relatively basic
technical knowledge to use it. In
addition, they will need to agree to
enable screen sharing.
The mismatching of participants
can be a source of frustration in any
study. Regardless of the reason for the
mismatch with your study, you will
need to determine how much of their
feedback is helpful and how much
should be dismissed.
Planning moderated sessions can be time
consuming. Those that occur between
people in different time zones is always
my biggest personal challenge. I’m
thankful for the World Clock meeting
planner ( http://www.timeanddate.com/
worldclock/ meeting.html). I use it at least
once a week to ensure that I am planning
meetings that make sense for most, if not
Doodle ( http://doodle.com/) is
another tool that I have found to be
irreplaceable in my quest to solve the
planning challenge. Doodle allows for
time-zone specification, so that when
I’m offering time blocks, people see
them in their local time zone.
Build time into the study to
compensate for any technical issues. I
typically add five to 15 minutes of time
to my study for technical considerations.
In some cases that means I will have
very little time for the actual study.
Always end the study when you said you
would. Participants have planned this
time for you; you need to respect their
schedule as well.
Send participants a calendar
invitation with the pertinent details
included in the invite, as well as an email
with additional details and a link to the
consent form. If possible, to further
improve attendance, ask participants to
communicate their plan to participate
in the remote study (location, Internet
access, admin rights, etc.).
Part of what makes usability studies
so effective is the ability for teams
to observe and learn firsthand about
the issues inherent in the system.
Observers who are collocated can be
in the same room as the moderator,
though background noise can be
challenging to minimize. For new
collocated observers, I typically
recommend a separate room and that
they stay on mute.
When the observers are themselves
remote, I always ask them to mute
themselves and to make sure that the
conference-line software has muted the
noise that accompanies people joining
and leaving the meeting. I also ask the
observers not to announce themselves.
I refer to a “colleague who is listening,”
and I avoid enumerating how many
people are watching or who is watching.
While there are ethical issues
with this approach, in most cases
introducing all the observers will
make the participant nervous and eat
up limited study time. As with other
studies, we provide the observers
with a guide and tips on note-taking,
and then hold a debrief session
with them. Frequently one or more
observers will take notes in a shared
It is not a question of if but rather when
technology issues will arise. While I
do ask participants to join the meeting
early and/or download the software
ahead of the session, it is the rare
participant who does so. Challenges
installing the minimal software
necessary for an online meeting is
typically the first issue.
Even when participants have
prepared, there are often last-minute
issues that were unanticipated.
Participants may be using accessibility
software to access the work, use an
alternative/old operating system, or
DOI: 10.1145/3038225 COPYRIGHT HELD B Y AUTHOR. PUBLICATION RIGHTS LICENSED TO ACM. $15.00