is when users’ hands cover the display, preventing them from seeing the graphics being guided.

Challenges. The gestures should be simple, temporally short, and natural. For a given set of tasks, users should have to remember at most only a few postures. Iconic representations of gesture-command associations may also help relieve users’ mental load.

Intuitiveness. The gesture types selected by interface developers should have a clear cognitive association with the functions they perform. For example, an open palm can represent a “stop” command, a closed fist with thumb up can represent “OK,” and a pointing finger can represent the direction to move an object. Few users are able to remember complex shapes and unnatural finger configurations. Intuitiveness is associated with other usability terms (such as learnability and “easy to remember”). Other factors affecting user-gesture choices are general knowledge, cultural environment, and linguistic capability. 51

Challenges. Intuitiveness is strongly associated with cultural background and experience. A gesture natural to one user may be unnatural to others. Moreover, Stern et al. 46 showed there is no consensus among users regarding gesture-function associations. This problem can be overcome by letting users decide which gesture best represents their intentions. The “Wizard of Oz” paradigm34 and analytical structured approaches51 help achieve this representation.

Comfort. Lexicon design should avoid gestures that require intense muscle tension over long periods, a syndrome commonly called “Gorilla arm.” Gestures must be concise and comfortable while minimizing stress on the hand. Awkward, repetitive postures can strain tissues and result in pressure within the carpal tunnel. Two types of muscular stress are found: static, the effort required to maintain a posture for a fixed amount of time, and dynamic, the effort required to move a hand through a trajectory.

Challenges. Measuring stress produced by hand gestures is very difficult. For stress-index measures, experiments vary from subjective questionnaires to electronic devices (such as electromyograms) that measure

Lexicon design
should avoid
gestures that
require intense
muscle tension
over long periods,
a syndrome
commonly called
“Gorilla arm.”

muscle activity. The main obstacle with physiological methods is that muscle potentials are highly variable within subjects and depend on external factors like positioning, temperature, and physiologic state. Instead, analytical approaches help assess stress based on the dynamics of musculoskeletal models.

Lexicon size and multi-hand systems. For sign languages (such as American Sign Language), hand-gesture-recognition systems must be able to recognize a large lexicon of both single-handed and two-handed gestures. For multi-touch systems, lexicon size plays a minor role. In either case, the challenge is to detect (and recognize) as many hands as possible.

Challenges. The different types of gestures to be recognized must be weighed against the system’s robustness. A classifier that recognizes a small number of gestures generally outperforms the same system trained on more gestures. The challenge for the vision algorithm is to select robust features and classifiers such that the system’s performance is barely affected by lexicon size. Multi-hand systems pose additional challenges (such as disambiguation of mutual hand occlusions and correctly associating hands and people).

Come as you are. 48 This phrase refers to an HCI design that poses no requirement on the user to wear markers, gloves, or long sleeves, fix the background, or choose a particular illumination. Many methods encumber the user in order to track and recognize gestures by standardizing the appearance of the hands (markers, gloves, long sleeves) but make interaction cumbersome. The challenge for the vision algorithm is to recognize hand gestures without requiring the user wear additional aids or being wired to a device.

Challenges. This flexibility constraint suggests a machine-vision-based solution that is not invasive. The drawback reveals itself with varied environments and user appearance. Assumptions about user characteristics and illumination affect system robustness. Near-IR illuminators can help. Far-IR cameras, ultrasonic, IR laser scanners, and capacitive imagers are also possible approaches for maintaining a system that lets users come as you are.