a similar manner, resulting in the widespread adoption of
such surface technologies.
2. oVERViEW oF oPERation
2. 1. imaging through an LCD using iR light
A key element in the construction of ThinSight is a device
known as a retro-reflective optosensor. This is a sensing element which contains two components: a light emitter and
an optically isolated light detector. It is therefore capable
of both emitting light and, at the same time, detecting the
intensity of incident light. If a reflective object is placed in
front of the optosensor, some of the emitted light will be
reflected back and will therefore be detected.
ThinSight is based around a 2D grid of retro-reflective
optosensors which are placed behind an LCD panel. Each
optosensor emits light that passes right through the entire
panel. Any reflective object in front of the display (such as a
fingertip) will reflect a fraction of the light back, and this can
be detected. Figure 2 depicts this arrangement. By using a suitably spaced grid of retro-reflective optosensors distributed uniformly behind the display it is therefore possible to detect any
number of fingertips on the display surface. The raw data generated is essentially a low resolution grayscale “image” of what
can be seen through the display, which can be processed using
computer vision techniques to support touch and other input.
A critical aspect of ThinSight is the use of retro-reflective
sensors that operate in the infrared part of the spectrum, for
three main reasons:
– Although IR light is attenuated by the layers in the LCD
panel, some still passes through the display.
5 This is
largely unaffected by the displayed image.
– A human fingertip typically reflects around 20% of incident IR light and is therefore a quite passable “reflective
– IR light is not visible to the user, and therefore does not
detract from the image being displayed on the panel.
Figure 2. the basic principle of thinSight. an array of retro-reflective
optosensors is placed behind an LCD. Each of these contains two
elements: an emitter which shines iR light through the panel; and
a detector which picks up any light reflected by objects such as
fingertips in front of the screen.
LCD panel Emitter Detector
2. 2. Further features of thinSight
ThinSight is not limited to detecting fingertips in contact with the display; any suitably reflective object will
cause IR light to reflect back and will therefore generate a
“silhouette.” Not only can this be used to determine the location of the object on the display, but also its orientation and
shape, within the limits of sensing resolution. Furthermore,
the underside of an object may be augmented with a visual
mark—a barcode of sorts—to aid identification.
In addition to the detection of passive objects via their
shape or some kind of barcode, it is also possible to embed
a very small infrared transmitter into an object. In this way,
the object can transmit a code representing its identity, its
state, or some other information, and this data transmission
can be picked up by the IR detectors built into ThinSight.
Indeed, ThinSight naturally supports bidirectional IR-based
data transfer with nearby electronic devices such as smartphones and PDAs. Data can be transmitted from the display to a device by modulating the IR light emitted. With a
large display, it is possible to support several simultaneous
bidirectional communication channels in a spatially multiplexed fashion.
Finally, a device which emits a collimated beam of IR light
may be used as a pointing device, either close to the display
surface like a stylus, or from some distance. Such a pointing
device could be used to support gestures for new forms of
interaction with a single display or with multiple displays.
Multiple pointing devices could be differentiated by modulating the light generated by each.
3. thE thinSiGht haRDWaRE
3. 1. the sensing electronics
The prototype ThinSight circuit board depicted in Figure
3 uses Avago HSDL-9100 retro-reflective infrared sensors.
These devices are especially designed for proximity sensing
—an IR LED emits infrared light and an IR photodiode generates a photocurrent which varies with the amount of incident
light. Both emitter and detector have a center wavelength of
A 7 × 5 grid of these HSDL-9100 devices on a regular
10mm pitch is mounted on custom-made 70 × 50mm
4-layer printed circuit board (PCB). Multiple PCBs can be
tiled together to support larger sensing areas. The IR detectors are interfaced directly with digital input/output lines on
a PIC18LF4520 microcontroller.
The PIC firmware collects data from one row of detectors at a time to construct a “frame” of data which is then
transmitted to the PC over USB via a virtual COM port. To
connect multiple PCBs to the same PC, they must be synchronized to ensure that IR emitted by a row of devices on
one PCB does not adversely affect scanning on a neighboring PCB. In our prototype we achieve this using frame and
row synchronization signals which are generated by one
of the PCBs (the designated “master”) and detected by the
Note that more information on the hardware can be
found in the full research publications.