One system designed to allow people fluent in ASL to communicate with non-signers is SignAloud, which was developed in 2016 by a pair of University of Washington undergraduate students. The system consists of a pair of gloves that are designed to recognize the hand gestures that correspond to words and phrases used in American Sign Language (ASL).

Worn by the signer, each glove is fitted with motion-capture sensors that record the position and movements of the hand wearing it, then sends that data to a central computer via a wireless Bluetooth link. The data is fed through various sequential statistical regressions, which are similar to a neural network, for processing. When the data matches an ASL gesture, the associated word or phrase is spoken through a speaker. The idea is to allow for real-time translation of ASL into spoken English.

Despite the promise of SignAloud, whose inventors received the $10,000 Lemelson-MIT Student Prize, there was significant criticism of the product from the deaf community, who complained that SignAloud did not capture the nuances of sign language, which relies on secondary signals such as eyebrow movements, shifts in the signer’s body, and motions of the mouth, to fully convey meaning and intent. Furthermore, strict word-for-word translations of ASL, like other languages, often results in an inaccurate translation, as each language requires sentence structure and context in order to make sense.

That has not stopped other companies from developing similar products, such as the BrightSign Glove, developed by Hadeel Ayoub as a relatively inexpensive (pricing is expected to be in the hundreds-of-dollars range) way to allow two-way communication between those who sign and those who do not. BrightSign’s technology is slightly different than SignAloud; users record and name their own gestures to correspond with specific words or phrases, thereby ensuring that the lack of facial cues or body motions will not impact meaning. As a result, BrightSign users can take advantage of a 97% accuracy rate when using the gloves.

BrightSign is readying several ver-
sions of the glove for commercializa-
tion, including a version aimed at chil-
dren, with a substantial wristband with
its own embedded screen and audio
output. Another version, targeted at
the adult deaf community, can send
translations directly to the wearer’s
smartphone, which can then enunciate
the words or phrases.
The company says it has about 700
customers on its preorder list, and is
trying to secure about $1.4 million in
capital from investors, which would
allow the company to fulfill all exist-
ing preorders.
Other tools are being developed to
address the technological challenges
of translating ASL to speech, although
the complexity of ASL and other sign
languages present significant tech-
nological challenges to handle these
tasks in real time, which is needed to
ensure smooth communication.
“There are several companies that
are developing software and databases,
including with the use of AI and ma-
chine learning, to create programs on
computers that can ‘read’ a person that
is signing in ASL,” Rosenblum says,
noting that these tools not only read
hand-signing, but also capture facial
cues and other visible markers. Using
cameras to capture these signs and
cues, the systems then use machine
learning to identify and recognize spe-
cific movements and gestures, and then
match them to specific words or phras-
es which can then be sent to a speech or
text generator that can be read or heard
by a non-signing individual.
“However, the challenge is that ev-
ery person signs with their own flair
and nuance, just like every person has
a different sound or inflection on how
they pronounce certain words,” Rosen-
blum says. To manage the variances in
the way people sign, videos of people
signing must be input and processed
by a machine learning algorithm to
train the system to account for these
stylistic variances. As such, the systems
need lots of time and data in order to
improve accuracy.
Another major issue is allowing peo-
ple who don’t sign to communicate in
real time with those who do sign. One
application that appears to be func-
tioning well enough for some users to
utilize today is Hand Talk. This app al-
lows a non-signer to input words and
phrases by speaking to the app located
on a deaf person’s phone. The app en-
gine translates the words in real time
into Libras, the sign language used
in Brazil. Then, an animated avatar
known as Hugo will begin signing on
the deaf person’s smartphone screen.
Unlike other apps that are using ma-
chine learning to train an algorithm,
Hand Talk’s founder Ronaldo Tenorio
and his team program thousands of
example sentences every month and
match them with three-dimensional (3D)
animations of sign language, including
Hugo’s facial expressions, which carry
meaning in sign language. Improve-
ments to the application are pushed out
through regular app updates.
According to the company, the app
handles six million monthly transla-
tions on Hand Talk, and has reached
one million downloads, approximately
one-sixth of Brazil’s deaf population.
Still, for applications that will be
useful across a wide range of languag-
es, cultures, and situations, developers
likely will need to use machine learn-
ing algorithms to learn all the possible
variations, nuances, and cadences of
conversational sign language. Further,
ASL and other sign languages are very
complex, with signs bleeding into one
another, anticipating the shape or lo-
cation of the following sign, which is
similar to how some spoken sounds
take on the characteristics of adja-
cent sounds. As such, Rosenblum
says, “the capacity or development of
computers being able to “read” the
zillions of variations of rendering ASL
is extremely difficult and probably will
take a decade to accomplish.”
“The challenge is that
every person signs
with their own flair
and nuance, just like
every person has
a different sound
or inflection on how
they pronounce
certain words.”