Communications of the ACM

of DARPA’s L2M program and a computer science professor at the University of Massachusetts, Amherst. “We will never be safe in a self-driving car without it,” she says. But it is just one of many necessary steps toward that goal. “It’s definitely not the end of the story,” she says.

There are five “pillars” of lifelong learning as DARPA broadly defines it, and synaptic plasticity falls into the first of these. The pillars are: continuous updating of memory, without catastrophic forgetting; recombinant memory, rearranging and recombining previously learned information toward future behavior; context awareness and context based modulation of system behavior; adoption of new behaviors through internal play, self-awareness, and self-simulations; and safety and security, recognizing whether something is dangerous and changing behavior accordingly, and ensuring security through a combination of strong constraints.

Siegelmann cites smart prosthe-ses as an example of an application of these techniques. She says the control software in an artificial leg could be trained via conventional backpropagation by its maker, then trained to the unique habits and characteristics of its user, and finally enabled to very quickly adapt to a situation it has not seen before, such as an icy sidewalk.

A computational neuroscientist,
Siegelmann says lifelong learning has
been a goal of AI researchers for many
years, but major advancements have
only recently become feasible, en-
abled by advancements in computer
power, new theoretical foundations
and algorithms, and a better under-
standing of biology. “In a few years,
much of what we call AI today won’t be
considered AI without lifelong learn-
ing,” she predicts.

Miconi’s team is now working on making learning more dynamic and sophisticated than it is in his test systems so far. One way to do that is to make the plasticity coefficients, now fixed as a design choice, themselves variable over the life of a system. “The plasticity of each connection can be determined at every point by the network itself,” he says. Such “ neuro-modulation” likely occurs in animal brains, he says, and that may be a key step toward the most flexible decision-making by AI systems.

Further Reading

Chang, O. and Lipson, H.

Neural Network Quine, Data Science Institute, Columbia University, New York, NY 10027, May 2018 https://arxiv.org/abs/1803.05859v3

Chen, Z. and Liu, B.

Lifelong Machine Learning, Second Edition,

Synthesis Lectures on Artificial Intelligence and Machine Learning, August 2018 https://www.morganclaypool. com/doi/10.2200/ S00832ED1V01Y201802AIM037

Hebb, D.

The Organization of Behavior: A Neuropsychological Theory, New York: Wiley & Sons, 1949 http://s-f-walker.org.uk/pubsebooks/pdfs/The_ Organization_of_Behavior-Donald_O._Hebb.pdf

Miconi, T., Clune, J., and Stanley, K.

Differentiable Plasticity: Training Plastic Neural Networks with Backpropagation, Proceedings of the 35th International Conference on Machine Learning (ICML

2018), Stockholm, Sweden, PMLR 80, 2018 https://arxiv.org/abs/1804.02464

Miconi, T.

Backpropagation of Hebbian Plasticity for Continual Learning,

NIPS Workshop on Continual Learning, 2016 https://github.com/ThomasMiconi/ LearningToLearnBOHP/blob/master/paper/ abstract.pdf

highly accurate results when classify-
ing different kinds of automobiles,
but when a new kind of car (a Tesla,
say) is seen, the system stumbles.
“You want it to recognize this new
car very quickly, without retraining,
which can take days or weeks. Also,
how do you know that something new
has happened?”
Artificial intelligence systems that
learn on the fly are not new. In “neu-
roevolution,” networks update them-
selves by algorithms that employ a
trial-and-error method to achieve a
precisely defined objective, such as
winning a game of chess. They require
no labeled training examples, only
definitions of success. “They go only
by trial and error,” says Uber’s Miconi.
“It’s a powerful, but a very slow, es-
sentially random, process. It would
be much better if, when you see a new
thing, you get an error signal that tells
you in which direction to alter your
weights. That’s what backpropagation
gets you.”

Military Apps

Miconi’s ideas represent just one of
a number of new approaches to self-
learning in AI. The U.S. Department of
Defense is pursuing the idea of synap-
tic plasticity as part of a broad family
of experimental approaches aimed at
making defense systems more accu-
rate, responsive, and safe. The U.S.
Defense Advanced Research Projects
Agency (DARPA) has established a
Lifelong Learning Machines (L2M)
program with two major thrusts, one
focused on the development of com-
plete systems and their components,
and the second on exploring learning
mechanisms in biological organisms
and translating them into computa-
tional processes. The goals are to en-
able AI systems to “learn and improve
during tasks, apply previous skills
and knowledge to new situations, in-
corporate innate system limits, and
enhance safety in automated assign-
ments,” DARPA says at its website.
“We are not looking for incremental
improvements, but rather paradigm-
changing approaches to machine
learning.”
Uber’s work with Hebbian plastic-
ity is a promising step toward lifelong
learning in neural networks, says Hava
Siegelmann, founder and manager