sophisticated IR illumination scheme. We have been experimenting with using an acrylic overlay on top of the LCD and
using IR LEDs for edge illumination. This would allow us to
sense multiple touch points using standard Frustrated Total
Internal Reflection (FTIR),
5 but not objects. We have, however, also experimented with a material called Endlighten
which allows this FTIR scheme to be extended to diffuse illumination, allowing both multitouch and object sensing with
far fewer IR emitters than our current setup. The overlay can
also serve the dual purpose of protecting the LCD from flexing under touch.
6. RELatED WoRK
The area of interactive surfaces has gained particular
attention recently following the advent of the iPhone and
Microsoft Surface. However, it is a field with over two decades
3 Despite this sustained interest there has been an
evident lack of off-the-shelf solutions for detecting multiple
fingers and/or objects on a display surface. Here, we summarize the relevant research in these areas and describe the
few commercially available systems.
6. 1. Camera-based systems
One approach to detecting multitouch and tangible input
is to use a video camera placed in front of or above the
surface, and apply computer vision algorithms for sensing. Early seminal work includes Krueger’s VideoDesk13
and the DigitalDesk,
26 which use dwell time and a microphone (respectively) to detect when a user is actually touching the surface. More recently, the Visual Touchpad17 and
C-Slate9 use a stereo camera placed above the display to
more accurately detect touch. The disparity between the
image pairs determines the height of fingers above the
surface. PlayAnywhere28 introduces a number of additional image processing techniques for front-projected
vision-based systems, including a shadow-based touch
detection algorithm, a novel visual bar code scheme,
paper tracking, and an optical flow algorithm for bimanual interaction.
Camera-based systems such as those described above
obviously require direct line-of-sight to the objects being
sensed which in some cases can restrict usage scenarios.
Occlusion problems are mitigated in PlayAnywhere by
mounting the camera off-axis. A natural progression is to
mount the camera behind the display. HoloWall18 uses IR
illuminant and a camera equipped with an IR pass filter
behind a diffusive projection panel to detect hands and
other IR-reflective objects in front of it. The system can accurately determine the contact areas by simply thresholding
the infrared image. TouchLight27 uses rear-projection onto a
holographic screen, which is also illuminated from behind
with IR light. A number of multitouch application scenarios
are enabled including high-resolution imaging capabilities.
Han5 describes a straightforward yet powerful technique
for enabling high-resolution multitouch sensing on rear-projected surfaces based on FTIR. Compelling multitouch
applications have been demonstrated using this technique.
The Smart Table22 uses this same FTIR technique in a tabletop form factor.
96 CommuniCationS oF thE aCm | DeCeMBeR 2009 | vOL. 52 | NO. 12
The Microsoft Surface and ReacTable12 also use rear-projection, IR illuminant and a rear mounted IR camera to
monitor fingertips, this time in a horizontal tabletop form-factor. These systems also detect and identify objects with
IR-reflective markers on their surface.
The rich data generated by camera-based systems provides extreme flexibility. However, as Wilson discusses28
this flexibility comes at a cost, including the computational demands of processing high resolution images, susceptibility to adverse lighting conditions and problems of
motion blur. However, perhaps more importantly, these
systems require the camera to be placed at some distance
from the display to capture the entire scene, limiting their
portability, practicality and introducing a setup and calibration cost.
6. 2. opaque embedded sensing
Despite the power of camera-based systems, the associated drawbacks outlined above have resulted in a number
of parallel research efforts to develop a non-vision-based
multitouch display. One approach is to embed a multi-touch sensor of some kind behind a surface that can have
an image projected onto it. A natural technology for this is
capacitive sensing, where the capacitive coupling to ground
introduced by a fingertip is detected, typically by monitoring
the rate of leakage of charge away from conductive plates or
wires mounted behind the display surface.
Some manufacturers such as Logitech and Apple have
enhanced the standard laptop-style touch pad to detect
certain gestures based on more than one point of touch.
However, in these systems, using more than two or three fingers typically results in ambiguities in the sensed data. This
constrains the gestures they support. Lee et al.
14 used capacitive sensing with a number of discrete metal electrodes
arranged in a matrix configuration to support multitouch
over a larger area. Westerman25 describes a sophisticated
capacitive multitouch system which generates x-ray-like
images of a hand interacting with an opaque sensing surface, which could be projected onto. A derivative of this work
was commercialized by Fingerworks.
DiamondTouch4 is composed of a grid of row and column antennas which emit signals that capacitively couple
with users when they touch the surface. Users are also
capacitively coupled to receivers through pads on their
chairs. In this way the system can identify which antennas
behind the display surface are being touched and by which
user, although a user touching the surface at two points can
produce ambiguities. The SmartSkin21 system consists of
a grid of capacitively coupled transmitting and receiving
antennas. As a finger approaches an intersection point,
this causes a drop in coupling which is measured to determine finger proximity. The system is capable of supporting
multiple points of contact by the same user and generating images of contact regions of the hand. SmartSkin and
Diamond Touch also support physical objects, but can only
identify an object when a user touches it. Tactex provide
another interesting example of an opaque multitouch sensor, which uses transducers to measure surface pressure at
multiple touch points.