review articles

ARTIFICIAL INTELLIGENCE (AI) systems have reached or exceeded human performance for many circumscribed tasks. As a result, they are increasingly deployed in mission-critical roles, such as credit scoring, predicting if a bail candidate will commit another crime, selecting the news we read on social networks, and self-driving cars. Unlike other mission-critical software, extraordinarily complex AI systems are difficult to test: AI decisions are context specific and often based on thousands or millions of factors. Typically, AI behaviors are generated by searching vast action spaces or learned by the opaque optimization of mammoth neural networks operating over prodigious amounts of training data. Almost by definition, no clear-cut method can accomplish these AI tasks.

Unfortunately, much AI-produced behavior is alien,
that is, it can fail in unexpected ways. This lesson is
most clearly seen in the performance
of the latest deep neural network im-
age analysis systems. While their accu-
racy at object-recognition on naturally
occurring pictures is extraordinary,
imperceptible changes to input im-
ages can lead to erratic predictions, as
shown in Figure 1. Why are these recog-
nition systems so brittle, making differ-
ent predictions for apparently identical
images? Unintelligible behavior is not
limited to machine learning; many AI
programs, such as automated planning
algorithms, perform search-based look
ahead and inference whose complexity
exceeds human abilities to verify. While
some search and planning algorithms
are provably complete and optimal, in-
telligibility is still important, because
the underlying primitives (for example,
search operators or action descrip-
tions) are usually approximations. 29
One can’t trust a proof that is based on
(possibly) incorrect premises.

Despite intelligibility’s apparent value, it remains remarkably difficult to specify what makes a system “ intelligible.” (We discuss desiderata for intelligible behavior later in this article.) In brief, we seek AI systems where it is clear what factors caused the system’s action, 24 allowing the users to predict how changes to the situation would have led to alternative behaviors, and permits effective control of

The Challenge
of Crafting
Intelligible
Intelligence
DOI: 10.1145/3282486

To trust the behavior of complex AI algorithms, especially in mission-critical settings, they must be made intelligible.

BY DANIEL S. WELD AND GAGAN BANSAL

key insights

˽ There are important technical and social reasons to prefer inherently intelligible AI models (such as linear models or GA2Ms) over deep neural models; furthermore, intelligible models often have comparable accuracy.

˽ When an AI system is based on an inscrutable model, it may explain its decisions by mapping those decisions onto a simpler, explanatory model using techniques such as local approximation and vocabulary transformation.

˽ Results from psychology show that explanation is a process, best thought of as a conversation between explainer and listener. We advocate increased work on interactive explanation systems that can respond to a wide range of follow-up questions.