a class of practical applications that
correspond to functions that, we now
know, are simple enough to allow
compact representations that can be
evaluated efficiently (again, without
the need for reasoning), and whose
estimation is within reach of current
thresholds for gathering data, com-
reach. The lack of such characteriza-
tion is a culprit of current misconcep-
tions about AI progress and where it
may lead us in the future.

What Just Happened in AI?

In my own quest to fully appreciate the progress enabled by deep learning, I came to the conclusion that recent developments tell us more about the problems tackled and the structure of our world than about neural networks per se. These networks are parameterized functions that are expressive enough to capture any relationship between inputs and outputs and have a form that can be evaluated efficiently. This has been known for decades and described at length in textbooks. What caused the current turn of events?

To shed some light on this question, let me state again what we have discovered recently. That is, some seemingly complex abilities that are typically associated with perception or cognition can be captured and reproduced to a reasonable extent by simply fitting functions to data, without having to explicitly model the environment or symbolically reason about it. While this is a remarkable finding, it highlights problems and thresholds more than it highlights technology, a point I explain next.

Every behavior, intelligent or not, can be captured by a function that maps inputs (environmental sensing) to outputs (thoughts or actions). However, the size of this function can be quite large for certain tasks, assuming the function can be evaluated efficiently. In fact, the function may have an unbounded size in general, as it may have to map from life histories. The two key questions then are the following: For tasks of interest, are the corresponding functions simple enough to admit a compact representation that allows mapping inputs to outputs efficiently, as in neural networks (without the need for reasoning)? And, if the answer is yes, are we currently able to estimate these functions from input-output pairs (labeled data)?

What has happened in AI recently
are three developments that bear di-
rectly on these questions: The first is
our improved ability to fit functions
to data, which has been enabled by
the availability of massive amounts
of labeled data; the increased com-
putational power we now have at
our hands; and the increasingly so-
phisticated statistical and optimiza-
tion techniques for fitting functions
(including new activation functions
and new/deeper network structures).
The second is that we have identified

Figure 1. Object recognition and localization in an image (ImageNet).

Figure 2. Fitting a simple function to data.

0246810
x

5 6 7 8 9 10

y