reasoning is required to compute the
function outputs from its inputs. The
main tool of this approach is the neural
network. Many college students have
exercised a version of it in a physics
or chemistry lab, where they fit simple
functions to data collected from vari-
ous experiments, as in Figure 2. The
main difference here is we are now em-
ploying functions with multiple inputs
and outputs; the structure of these
functions can be quite complex; and
the problems being tackled are ones
we tend to associate with perception or
cognition, as opposed to, say, estimat-
ing the relationship between volume
and pressure in a sealed container.d
The main observation in AI recently
is that the function-based approach
can be quite effective at certain AI
tasks, more so than the model-based
approach or at least earlier attempts at
using this approach. This has surprised
not only mainstream AI researchers,
who mainly practice the model-based
approach, but also machine learning
researchers who practice various ap-
proaches, of which the function-based
approach is but one.e This has had
many implications, some positive and
some giving grounds for concern.
On the positive side is the increasing number of tasks and applications
now within reach, using a tool that can
be very familiar to someone with only
a broad engineering background, particularly one accustomed to estimating functions and using them to make
predictions. What is of concern, however, is the current imbalance between
exploiting, enjoying, and cheering
this tool on the one hand and thinking
about it on the other. This thinking is
not only important for realizing the full
potential of the tool but also for scientifically characterizing its potential
d This is also called the “curve-fitting” approach. While the term “curve” highlights the
efficient evaluation of a function and captures
the spirit of the function-based approach, it
underplays the complex and rich structure of
functions encoded by today’s (deep) neural
networks, which can have millions if not billions of parameters.
e Machine learning includes the function-based
approach but has a wide enough span that it
overlaps with the model-based approach; for
example, one can learn the parameters and
structure of a model but may still need non-trivial reasoning to obtain answers from the
learned model.
that does not require explicit model-
ing or sophisticated reasoning is suf-
ficient for reproducing human-level
intelligence. This dilemma is further
amplified by the observation that re-
cent developments did not culminate
in a clearly characterized and profound
scientific discovery (such as a new
theory of the mind) that would nor-
mally mandate massive updates to the
AI curricula. Scholars from outside AI
and computer science often sense this
dilemma, as they complain they are
not receiving an intellectually satisfy-
ing answer to the question: “What just
happened in AI?”
The answer lies in a careful assess-
ment of what we managed to achieve
with deep learning and in identifying
and appreciating the key scientific out-
comes of recent developments in this
area of research. This has unfortunate-
ly been lacking to a great extent. My
aim here is to trigger such a discussion,
encouraged by the positive and curious
feedback I have been receiving on the
thoughts expressed in this article.
Background
To lay the ground for the discussion, I
first mark two distinct approaches for
tackling problems that have been of
interest to AI. I call the first one “
model-based” and the second “
function-based.” Consider the object-recogni-tion and -localization task in Figure 1.
To solve it, the model-based approach
requires one to represent knowledge
about dogs and hats, among other
things, and involves reasoning with
such knowledge. The main tools of
the approach today are logic and probability (mathematical modeling more
generally) and can be thought of as
the “represent-and-reason”c approach
originally envisioned by the founders
of AI. It is also the approach normally
expected, at some level, by informed
members of the scientific community.
The function-based approach, on the
other hand, formulates this task as a
function-fitting problem, with function inputs coming directly from the
image pixels and outputs corresponding to the high-level recognitions we
seek. The function must have a form
that can be evaluated efficiently so no
c This term might be likened to what has been
called “good old-fashioned AI.”
In my own quest
to fully appreciate
the progress
enabled by deep
learning, I came
to the conclusion
that recent
developments tell
us more about the
problems tackled
and the structure
of our world than
about neural
networks per se.