a class of practical applications that
correspond to functions that, we now
know, are simple enough to allow
compact representations that can be
evaluated efficiently (again, without
the need for reasoning), and whose
estimation is within reach of current
thresholds for gathering data, com-
reach. The lack of such characteriza-
tion is a culprit of current misconcep-
tions about AI progress and where it
may lead us in the future.
What Just Happened in AI?
In my own quest to fully appreciate the
progress enabled by deep learning,
I came to the conclusion that recent
developments tell us more about the
problems tackled and the structure of
our world than about neural networks
per se. These networks are parameterized functions that are expressive
enough to capture any relationship
between inputs and outputs and have
a form that can be evaluated efficiently.
This has been known for decades and
described at length in textbooks. What
caused the current turn of events?
To shed some light on this question,
let me state again what we have discovered recently. That is, some seemingly
complex abilities that are typically associated with perception or cognition
can be captured and reproduced to
a reasonable extent by simply fitting
functions to data, without having to explicitly model the environment or symbolically reason about it. While this
is a remarkable finding, it highlights
problems and thresholds more than it
highlights technology, a point I explain
next.
Every behavior, intelligent or not,
can be captured by a function that
maps inputs (environmental sensing)
to outputs (thoughts or actions). However, the size of this function can be
quite large for certain tasks, assuming
the function can be evaluated efficiently. In fact, the function may have an unbounded size in general, as it may have
to map from life histories. The two key
questions then are the following: For
tasks of interest, are the corresponding functions simple enough to admit
a compact representation that allows
mapping inputs to outputs efficiently,
as in neural networks (without the
need for reasoning)? And, if the answer
is yes, are we currently able to estimate
these functions from input-output
pairs (labeled data)?
What has happened in AI recently
are three developments that bear di-
rectly on these questions: The first is
our improved ability to fit functions
to data, which has been enabled by
the availability of massive amounts
of labeled data; the increased com-
putational power we now have at
our hands; and the increasingly so-
phisticated statistical and optimiza-
tion techniques for fitting functions
(including new activation functions
and new/deeper network structures).
The second is that we have identified
Figure 1. Object recognition and localization in an image (ImageNet).
Figure 2. Fitting a simple function to data.
0246810
x
5
6
7
8
9
10
y