Technology | DOI: 10.1145/2800498 Gregory Mone
How computer systems detect the internal emotional states of users.
focus on visual input, and on the face
in particular. Until recently, simply
recognizing and tracking faces in
a crowded, dimly lit environment
such as a Super Bowl party was a serious challenge. Early systems could
be confused by something as pedestrian as a beard. Today’s software can
delineate contours in the lips, eyes,
nose, and more, and it can do so even
in poor lighting conditions, according to Akshay Ashtana, a facial recognition researcher at Seeing Machines
in Canberra, Australia.
Emotient and Affectiva benefit from
this technology, since it allows them
to pick out multiple faces in a single
crowded scene, but before they focus
their systems on those targets, they
have to train them to recognize the different expressions and emotions. Both
technologies are based on psychologist
Paul Ekman’s facial action coding system, or FACS, which details the involuntary muscle movements we generate
in response to different stimuli, and
how these subtle cues are linked to different emotional states.
Generating Quality Data
Emotient employed trained consultants to produce and analyze hundreds
DURING THE LAST Super Bowl, Emotient, a San Diego, CA- based company, hosted an unusual party. Emotient re- cruited 30 volunteers to a local bar to watch the game, eat, and drink.
As the annual spectacle progressed on
two televisions, a camera attached to
each flat screen monitored the viewers.
Behind the scenes, the Emotient Analytics system identified and tracked each
face within the camera’s view, then determined the changing emotional state
of each individual over time. Whether
they were amused, ambivalent, or surprised, the system matched those reactions to what was happening on screen,
determining which commercials seemed
to bore the viewers and which ones they
liked enough to share. “We were able to
predict which advertisements were likely
to go viral based on facial behavior,” says
lead scientist Marian Bartlett.
Both the Emotient technology and
a similar system from Waltham, MA-based Affectiva are products of the
burgeoning field of affective computing, in which researchers are developing systems that estimate the internal
emotional state of an individual based
on facial expressions, vocal inflections,
gestures, or other physiological cues.
Affectiva and Emotient are catering to
advertising and market research companies, but affective computing stands
to impact a number of areas. The technique could lead to online learning
systems that notice when a student
is growing frustrated with a problem,
healthcare applications that measure
how a depressed patient is responding
to a new medication, or in-home assistance robots that closely monitor the
emotional state of the elderly.
The field has been around for two
decades; Affectiva co-founder and Massachusetts Institute of Technology
professor Rosalind Picard coined the
term “affective computing” in the mid-
1990s. Yet experts say a combination of
Focusing on Faces
improved data, increased processing
power, and more robust machine learn-
ing techniques has led to significant
recent advances. “Affective comput-
ing has really exploded in the last five
years,” says signal processing expert
Shrikanth Narayanan of the University
of Southern California.
When people communicate with each
other, we study each other’s vocal intonations, gestures, facial expressions,
and posture, each of which gives us some
information about the other person’s internal state—whether they are engaged,
distracted, or annoyed. In some cases,
the expressions are contradictory; the
fact that a child is smiling, for example,
might seem to indicate she is happy. Yet
Narayanan’s work has shown how computers trained to pick up clues in the
pitch, intonation, or rhythm of a child’s
speech can uncover hidden emotions
lurking behind that smile.
This sort of detection is natural for
humans; we track multiple cues to
guess what someone is really thinking
or feeling. “We’re implicitly integrat-
ing all these pieces of information,”
Narayanan says. “We’re not picking
one or the other.”
At this point, though, the most
popular affective computing systems
Quantifying clues to human emotions.