review articles
ARTIFICIAL INTELLIGENCE (AI) systems have reached or
exceeded human performance for many circumscribed
tasks. As a result, they are increasingly deployed in
mission-critical roles, such as credit scoring, predicting
if a bail candidate will commit another crime, selecting
the news we read on social networks, and self-driving cars. Unlike other mission-critical software,
extraordinarily complex AI systems are difficult to
test: AI decisions are context specific and often based
on thousands or millions of factors. Typically, AI
behaviors are generated by searching vast action spaces
or learned by the opaque optimization of mammoth
neural networks operating over prodigious amounts of
training data. Almost by definition, no clear-cut method
can accomplish these AI tasks.
Unfortunately, much AI-produced behavior is alien,
that is, it can fail in unexpected ways. This lesson is
most clearly seen in the performance
of the latest deep neural network im-
age analysis systems. While their accu-
racy at object-recognition on naturally
occurring pictures is extraordinary,
imperceptible changes to input im-
ages can lead to erratic predictions, as
shown in Figure 1. Why are these recog-
nition systems so brittle, making differ-
ent predictions for apparently identical
images? Unintelligible behavior is not
limited to machine learning; many AI
programs, such as automated planning
algorithms, perform search-based look
ahead and inference whose complexity
exceeds human abilities to verify. While
some search and planning algorithms
are provably complete and optimal, in-
telligibility is still important, because
the underlying primitives (for example,
search operators or action descrip-
tions) are usually approximations.
29
One can’t trust a proof that is based on
(possibly) incorrect premises.
Despite intelligibility’s apparent
value, it remains remarkably difficult
to specify what makes a system “
intelligible.” (We discuss desiderata for intelligible behavior later in this article.)
In brief, we seek AI systems where it
is clear what factors caused the system’s action,
24 allowing the users to
predict how changes to the situation
would have led to alternative behaviors, and permits effective control of
The Challenge
of Crafting
Intelligible
Intelligence
DOI: 10.1145/3282486
To trust the behavior of complex AI algorithms,
especially in mission-critical settings,
they must be made intelligible.
BY DANIEL S. WELD AND GAGAN BANSAL
key insights
˽ There are important technical and social
reasons to prefer inherently intelligible
AI models (such as linear models
or GA2Ms) over deep neural models;
furthermore, intelligible models often
have comparable accuracy.
˽ When an AI system is based on an
inscrutable model, it may explain its
decisions by mapping those decisions
onto a simpler, explanatory model using
techniques such as local approximation
and vocabulary transformation.
˽ Results from psychology show that
explanation is a process, best thought
of as a conversation between explainer
and listener. We advocate increased
work on interactive explanation systems
that can respond to a wide range of
follow-up questions.