Communications of the ACM

to, and learn from, the interaction as
they jointly determine its content and
quality through real-time contingent
and reciprocal coaction.”
Another important factor in the ac-
ceptance of a virtual human face is its
visual quality. However, realistically
simulating a face—even when still—
has proved to be formidable.

Challenges of Modeling
the Human Face

The way a digital face moves and appears can cause unwanted effects. Rather than aiding the appearance of life, a partially realistic solution can elicit a negative response—the uncanny valley effect. This response is thought to be triggered by any number of non-expected responses, alarming the viewer’s perceptual system. 19 To avoid this response, many factors must be taken into account, including the physical appearance of the face and the eye-gaze movement of the skin, any of which can trigger a form of dissonance that interferes with the affinity of the perceived face.

Appearance. The ability to “read” faces is so important that several different parts of the brain play a role in face perception. We are sensitive to many factors that act as signs of health and vitality. People often refer to someone being “as white as a sheet,” “red faced,” or “sickly looking”; it is thus important to render physically plausible healthy skin with correct surface properties, detail, and subsurface scattering of light that provides diffuse properties of skin.

This challenge has been approached
in two broadly different ways. First, by
using “image-based methods” that
sample the face under different light-
ing and viewing conditions9 and then
render the face through a combina-
tion of weighted image-blended sets,
photogrammetry, and/or image pro-
jection. Second, by using “parametric
methods” that fit the captured data to
a face and material model used during
rendering, allowing for more flexibility
but at the cost of potentially increased
rendering complexity. 16 Given the con-
straints producing imagery fast enough
for user interaction, adding further to
the complexity of achieving an effec-
tive interactive face, a simplified imple-
mentation of the second approach is
dog “Silas” developed by Blumberg3 us-
ing sophisticated ethological models
to simulate how animals are able to or-
ganize and adapt.

These and similarly inspired works are important on many levels, as they explain how behavior can emerge, made observable through animation with constraints. Blumberg3 suggested for a creature to appear alive it must react, have goals, make choices, convey its intentionality, emotionally respond to events, adapt, and vary its movement and response.

For autonomous animation of the face in real time, Terzopolous and Lee37 developed a physics-based face model driven by a basic behavioral animation model. Despite this pioneering work, few other virtual human studies have focused on this lower level of detail in real-time facial animation.

Most research in building autonomous human agents has been as “embodied conversational agents” (such as in Allbeck1 and in Cassell7) at a generally more phenomenologi-cal and higher level, not specifically focused on the subtler details of facial expression and nonverbal behavior; Vinciarelli et al. 39 included a survey of social-signal processing in computer interaction. Simulating these signals is getting greater attention from the interdisciplinary “intelligent virtual agent” community, 5 exploring agents that are capable of real-time perception, cognition, and actions in the social environment; Marsella and Gratch21 discussed simulations of psychological theories of emotion. Emo-tion-oriented APIs (such as SEMAINE) have been developed. 33 And Scherer32 showed how cumulative effects of sequential checks of an eliciting event, mediated by autonomic and somatic components, might combine to create compound facial expressions.

Much of the work on virtual humans has an unfortunately robotic “feeling,” particularly with facial interaction. This is possibly due to most virtual human models not focusing on the microdynamics of expression or on facial realism. These microdynamics are considered particularly critical in learning contexts. Rohlfling and Deak27 stated: “When infants learn in a social environment, they do not simply pick up information passively. They respond