DOI: 10.1145/3097462 COP YRIGH T HELD B Y AU THOR
INTERACTIONS.ACM.ORG 8 INTERACTIONS JULY–AUGUST2017
she needs to attend to numerous features
and social interaction.
• In certain training situations,
affective capabilities are essential to
carrying out the task, such as in therapy,
which can be more physical, as in limb
injuries, or more mental, as in autism
disorder and social phobias, or both, as
in cases such as stroke rehabilitation.
In several of these situations—for
example, mental disorders such as
autism, anxiety, and social phobias—
the patient practices social interaction
while monitoring how they recognize or
respond to emotional situations.
• Well-being examples such as
physical exercise and meditation
(Figure 2). Affective interaction here
can motivate physical exercise or
monitor psychophysiological states
such as engagement or relaxation.
1. Ravaja, N., Harjunen, V., Ahmed, I.,
Jacucci, G., and Spapé, M. M. Feeling
touched: Emotional modulation of
somatosensory potentials to interpersonal
touch. Scientific Reports 7 (2017), 40504.
2. Harjunen, V.J., Spapé, M., Ahmed, I.,
Jacucci, G., and Ravaja, N. Individual
differences in affective touch: Behavioral
inhibition and gender define how
an interpersonal touch is perceived.
Personality and Individual Differences 107
3. Spapé, M.M., Hoggan, E.E., Jacucci, G.,
and Ravaja, N. The meaning of the virtual
Midas touch: An ERP study in economic
decision making. Psychophysiology 52, 3
4. Ahmed, I., Harjunen, V., Jacucci, G.,
Hoggan, E., Ravaja, N., and Spapé, M.M.
Reach out and touch me: Effects of four
distinct haptic technologies on affective
touch in virtual reality. Proc. of the 18th
ACM International Conference on Multimodal
Interaction. ACM, 2016, 341–348.
5. Affectiva; http://www.affectiva.com/
6. Kosunen, I., Salminen, M., Järvelä, S.,
Ruonala, A., Ravaja, N., and Jacucci, G.
Rela World: Neuroadaptive and immersive
virtual reality meditation system. Proc. of
the 21st International Conference on Intelligent
User Interfaces. ACM, 2016, 208–217.
Giulio Jacucci is a professor in computer
science at the University of Helsinki and
founder of Multi Taction ( www.multitaction.
com). His research interests include multimodal
interaction; physiological, tangible, and
ubiquitous computing; search and information
discovery; as well as behavioral change.
It can be used to simulate a social
interaction at a table where mediated
multimodal affect can be studied while
an avatar touches the user’s hand, at the
same time delivering a facial expression.
The user recognizes the virtual hand
in Figure 1A as her own hand, as it is
synchronized in real time.
This setup can be used for a number
of training, entertainment, or therapy
purposes. For example, a recent product
applies VR for treating anxiety patients.
Recent studies have evaluated the
impact in training autism spectrum
disorder patients to apply this to dealing
with anxiety. In our own recent study,
we used the same setup for an air
hockey game. The haptics simulated the
hitting of the puck, and the emotional
expression of the avatar allowed us to
study effects on players’ performance
and experience of the game.
VR devices, applications, and content
are emerging quickly. An important
feature in the future will be the
affective capability of the environment,
including the recognition and synthesis
A variety of research challenges exist
for affective interaction:
• Techniques in recognizing users’
emotions from easily deployable
sensors, including the fusion of signals.
Physiological computing is advancing
fast in research and in commercial
products. Vision-based solutions that
track facial expression have recently
seen success; physiological-based sensors
could soon follow suit.
• Synthesis of affect utilizing
multiple modalities, as exemplified
here. For example, combining touch
and facial expression, but considering
also speech and its acoustic features
and other nonverbal cues. How
to ensure that these multimodal
expressions are generally valid and can
be generated uniquely.
While these are challenges,
the potential application fields are
numerous and replete with emerging
evidence of their relevance:
• Training in, for example, emergency
or disaster situations, but in principle
in any setting where a learner needs to
simulate a task in an environment where
Now Available via
Did you know that you can
now order many popular
ACM conference proceedings
Institutions, libraries and
individuals can choose
from more than 100 titles
on a continually updated
list through Amazon, Barnes
& Noble, Baker & Taylor,
Ingram and NACSCORP:
CHI, KDD, Multimedia,
SIGIR, SIGCOMM, SIGCSE,
and many more.
For available titles and
ordering info, visit:
Now Available via