is when users’ hands cover the display, preventing them from seeing the
graphics being guided.
Challenges. The gestures should be
simple, temporally short, and natural.
For a given set of tasks, users should
have to remember at most only a few
postures. Iconic representations of
gesture-command associations may
also help relieve users’ mental load.
Intuitiveness. The gesture types selected by interface developers should
have a clear cognitive association with
the functions they perform. For example, an open palm can represent
a “stop” command, a closed fist with
thumb up can represent “OK,” and a
pointing finger can represent the direction to move an object. Few users
are able to remember complex shapes
and unnatural finger configurations.
Intuitiveness is associated with other
usability terms (such as learnability
and “easy to remember”). Other factors
affecting user-gesture choices are general knowledge, cultural environment,
and linguistic capability. 51
Challenges. Intuitiveness is strongly
associated with cultural background
and experience. A gesture natural to
one user may be unnatural to others.
Moreover, Stern et al. 46 showed there
is no consensus among users regarding gesture-function associations. This
problem can be overcome by letting
users decide which gesture best represents their intentions. The “Wizard of
Oz” paradigm34 and analytical structured approaches51 help achieve this
Comfort. Lexicon design should
avoid gestures that require intense
muscle tension over long periods, a
syndrome commonly called “Gorilla
arm.” Gestures must be concise and
comfortable while minimizing stress
on the hand. Awkward, repetitive postures can strain tissues and result in
pressure within the carpal tunnel. Two
types of muscular stress are found:
static, the effort required to maintain a
posture for a fixed amount of time, and
dynamic, the effort required to move a
hand through a trajectory.
Challenges. Measuring stress produced by hand gestures is very difficult. For stress-index measures,
experiments vary from subjective questionnaires to electronic devices (such
as electromyograms) that measure
over long periods,
muscle activity. The main obstacle with
physiological methods is that muscle
potentials are highly variable within
subjects and depend on external factors like positioning, temperature, and
physiologic state. Instead, analytical
approaches help assess stress based
on the dynamics of musculoskeletal
Lexicon size and multi-hand systems. For sign languages (such as
American Sign Language), hand-gesture-recognition systems must be able
to recognize a large lexicon of both single-handed and two-handed gestures.
For multi-touch systems, lexicon size
plays a minor role. In either case, the
challenge is to detect (and recognize)
as many hands as possible.
Challenges. The different types of
gestures to be recognized must be
weighed against the system’s robustness. A classifier that recognizes a
small number of gestures generally
outperforms the same system trained
on more gestures. The challenge for
the vision algorithm is to select robust
features and classifiers such that the
system’s performance is barely affected by lexicon size. Multi-hand systems
pose additional challenges (such as
disambiguation of mutual hand occlusions and correctly associating hands
Come as you are. 48 This phrase refers to an HCI design that poses no
requirement on the user to wear markers, gloves, or long sleeves, fix the background, or choose a particular illumination. Many methods encumber the user
in order to track and recognize gestures
by standardizing the appearance of the
hands (markers, gloves, long sleeves)
but make interaction cumbersome.
The challenge for the vision algorithm
is to recognize hand gestures without
requiring the user wear additional aids
or being wired to a device.
Challenges. This flexibility constraint
suggests a machine-vision-based solution that is not invasive. The drawback
reveals itself with varied environments
and user appearance. Assumptions
about user characteristics and illumination affect system robustness. Near-IR illuminators can help. Far-IR cameras, ultrasonic, IR laser scanners, and
capacitive imagers are also possible
approaches for maintaining a system
that lets users come as you are.