systems, for instance multi-user interaction or even gaming, where the user in front of the camera is the active user. Additionally, it could add a “smart” aspect to existing devices, such as making any smart TV gesture-enabled (Figure 7).

Thermal biometrics. A body of recent work recognizes the need for a more transparent usable authentication mechanism. To be truly usable, authentication should be seamless and require no explicit actions from users. Biometric authentication is a promising approach because biometric characteristics can be used to simultaneously identify and authenticate a user. The ability of thermal cameras to detect subtle temperature changes enables the unique vein patterns on the back of our hands to be captured. We investigated using veins on the back of the hand for contactless and seamless authentication (Figure 8). It is invariant to changes in how the hand is posed as well as changes to the environment. Being accurate and invariant, vein-based authentication has the potential to seamlessly authenticate users of desktop computers and tabletops.

The dark side of thermal imaging.

As the writer David Wong once said, “New technology is not good or evil in and of itself. It’s all about how people choose to use it.” Although thermal imaging enables seamless authentication and novel interactive techniques based on the heat traces, it raises a threat in the form of thermal attacks. As shown in Figure 9, the heat traces could be extracted from smartphone screens and used to infer the PIN or pattern used for authentication. In our exploration [ 3], we found that the most-used smartphone authentication techniques, namely PINs and patterns, could be uncovered with up to a 100 percent success rate, even 30 seconds after authentication.

A window into our mind. A technology with the capability of sensing and inferring physiological signals has the ability to provide a window into our mind that can be used to adapt system behavior accordingly. Internal-state information is largely invisible from the outside of a user’s brain, and introspection often fails to reason about it in an unbiased and objective way.

However, our emotional state influences
our body temperature, in particular our
facial temperature (Figure 10), as the
face contains critical blood vessels
connected to our autonomic nervous
system. Using thermal imaging, these
cues give glimpses into our internal
states, such as our cognitive load.

Thermal cameras overcome the limitations of the contact sensors used in related research. It is also more robust than other contactless approaches, since the temperature signature is more resistant to conscious manipulation [ 4, 5].

Thermal imaging technologies.

Thermal imaging has entered the commercial market and is available in different forms, such as integrated into smartphones and as attachments to phone or USB cameras (Figure 11).

Figure 2. (left) RGB view; (right) equivalent thermal camera view.

Figure 4. Traces of object interaction. Figure 3. Step traces after the user is gone.

Figure 5. (left) RGB reflection behavior; (right) thermal mirror-like reflection for the same surface.

Figure 7. Potential use cases for thermal reflection interaction.

Figure 8. Veins from the back of the hand captured by a thermal camera.

Figure 9. Heat traces for sample PIN and pattern.

Figure 6. (left) The field of view (FOV) of the camera in gray. (right) The extended interaction space due to thermal reflection, highlighted in colors.