acceptance in 2012, after Hinton and
two students used deep neural nets to
win the ImageNet challenge, identifying objects in a set of photos at a rate
far better than that of any of their competitors. Since then, the field has embraced the technology, which has also
seen breakthroughs in speech recognition and natural language processing,
and could help make self-driving vehicles more reliable.
LeCun says theories about why
neural nets would not work—that the
training algorithms would get stuck
in the extreme values of mathematical
functions known as local minima—fell
to real-world experience. “In the end,
what people were convinced by were
not theorems; they were experimental
results,” he says. Even though there
were local minima, those bad enough
for an optimization algorithm to get
stuck were relatively rare. It turned
out that if the neural nets were just big
enough for the problem they were trying to solve, they could get stuck, but
if they were larger, they became more
efficient at optimization. “You make
those networks bigger and bigger and
they work better and better,” LeCun
says.
Working both together and independently, the three made important
contributions to neural networks.
Among their several discoveries, Hinton helped to develop backpropagation, an algorithm that calculates error
at the output of the network and propagates the results backward toward the
input, allowing the machine to improve
its accuracy. LeCun developed convolutional neural networks, which replicate
feature detectors across space and are
more efficient for image and speech
recognition.
Another development that helps the
system learn more effectively involves
randomly turning off some of the neurons about half of the time, introducing some noise into the network. Bengio says there is noise and randomness
in the way living neurons spike, and
something about that makes the system better at dealing with variations
in input patterns, which is key to making the system useful. “You want to be
good at doing the things you haven’t
yet seen, things that might be somewhat different from the training data,”
Hinton says.
Bengio came up with word embeddings, patterns of neuron activation
that represent word symbols, thereby
expanding exponentially the system’s
ability to express meanings and making it possible to process text and translate it from one language to another.
Hinton explains that the embeddings
make it easier for the system to reason
by analogy, rather than by following a
logical set of rules; he believes that is
more like how the human brain works.
The brain evolved to use patterns of
neural activity to perform perception
and movement, and that makes it more
suited to reasoning by analogy rather
than logic, he argues.
In fact, artificial intelligence remains limited compared to human intelligence. “Machines are still very, very
stupid,” LeCun says. “The smartest AI
systems today have less common sense
than a house cat.” Though they excel
at recognizing patterns, neural networks have no knowledge of how the
world works, and computer scientists
have not yet figured out how to give it
to them. Humans learn to generalize
from a very small number of samples,
while neural networks require vast sets
of training data. In fact, Hinton says, it
was the growth in available datasets,
along with faster processors, that led
to the “phase shift” from neural networks being a curiosity to a practical
approach.
There are hundreds of useful tasks
neural networks can accomplish just
by using their current pattern recog-
nition capabilities, Hinton says, from
predicting earthquake aftershocks to
offering better medical diagnoses on
the basis of hundreds of thousands of
examples. But to give machines a more
general intelligence that could solve
different types of problems or accom-
plish multiple tasks will require sci-
entists to come up with new concepts
about how learning works, Bengio
says. “It might take a very long time be-
fore we reach human-level AI,” he says.
Meanwhile, society has to have
more discussion about how to use artificial intelligence appropriately. Hinton worries about how autonomous
intelligent weapons systems might be
misused, for instance. LeCun says that
without adequate political and legal
protections, governments could use
the systems to track people and try to
control their behavior, or corporations
might rely on AI to make decisions but
ignore bias in their algorithms.
To address some of these worries,
Bengio took part in a group that last December issued the Montreal Declaration for a Responsible Development of
Artificial Intelligence, which outlines
principles that they say should be used
in pushing the technology forward.
“We’re building stronger and stronger
technology based on the premises of
science, but the organization of society
and their collective wisdom isn’t keeping up fast enough. The solution may
not be in some new theorem or some
new algorithm,” he says.
With such concerns in mind, Hinton says he will donate a portion of his
share of the $1-million Turing Award
prize money to the humanities at the
University of Toronto. “If we have science without the humanities to help
guide the political process, then we’re
all in trouble,” he says. LeCun says he
will likely make a donation to NYU, and
Bengio says he’s considering some environmental causes.
Based on their experiences as academic heretics who turned out to be
right, they advise young computer scientists to stick to their convictions. “If
someone tells you your intuitions are
wrong, there are two possibilities,”
Hinton says. “One is you have bad intuitions, in which case it doesn’t matter what you do, and the other is you
have good intuitions, in which case you
should follow them.”
Neil Savage is a science and technology writer based in
Lowell, MA, USA.
© 2019 ACM 0001-0782/19/6 $15.00
“Machines are still
very, very stupid,”
LeCun says.
“The smartest AI
systems today have
less common sense
than a house cat.”