Nnews

P H O T O C O P Y R I G H T © , P I T T S B U R G H P O S T - G A Z E T T E , 2 0 1 7 , A L L R I G H T S R E S E R V E D . R E P R I N T E D W I T H P E R M I S S I O N .

IN A 2016 video, Saqib Shaikh, a Microsoft Research software ngineer, walks out of Lon- don’s Clapham Station Under- ground stop, turns, and crosses a street, then stops suddenly when he hears an unexpected noise. Shaikh, who lost his sight when he was seven years old and walks with the aid of the standard white cane, reaches up and swipes the earpiece of his glasses.

The video then shifts to the view
from his eyewear, a pair of smart
glasses that capture high-quality still
images and videos. That simple swipe
instructed the glasses, an experimen-
tal prototype designed by a company
called Pivothead, to snap a still photo.
Microsoft software analyzed the pic-
ture, then translated the findings into
auditory feedback. Through the smart
glasses, which include a small speak-
er, Shaikh hears the results from an
automated voice: “I think it’s a man
jumping in the air doing a trick on a
skateboard.”
The Pivothead smart glasses and
Microsoft AI technology belong to a
broader class of what have become
known as sensory substitution tech-
nologies, apps and devices that col-
lect visual, auditory, and in some cases
Feeling Sounds,
Hearing Sights

A new wave of sensory substitution devices work to assist people who are blind or deaf.

Science | DOI: 10.1145/3157075 Gregory Mone

Chieko Asakawa, who is blind, uses the NavCog app, which she helped develop, to find her way on the campus of Carnegie Mellon University.