we have a seed test input, the image of a red car, which both
DNNs identify as a car as shown in Figure 6a.
DeepXplore tries to maximize the chances of finding differential behavior by modifying the input, that is, the image
of the red car, towards maximizing its probability of being
classified as a car by one DNN but minimizing corresponding probability of the other DNN. DeepXplore also tries to
cover as many neurons as possible by activating (i.e., causing a neuron’s output to have a value greater than a threshold) inactive neurons in the hidden layer. We further add
domain-specific constraints (e.g., ensure the pixel values are
integers within 0 and 255 for image input) to make sure that
the modified inputs still represent real-world images. The
joint optimization algorithm will iteratively perform a gradient ascent to find a modified input that satisfies all of the
goals described above. DeepXplore will eventually generate
a set of test inputs where the DNNs’ outputs differ, for example, one DNN thinks it is a car, whereas the other thinks it is
a face as shown in Figure 6b.
4. METHODOLOGY
In this section, we provide a brief technical description of
our algorithm. The details can be found in the original
paper. First, we define and explain the concepts of neuron
coverage and gradient for DNNs. Next, we describe how the
testing problem can be formulated as a joint optimization
problem. Finally, we provide the gradient-based algorithm
for solving the joint optimization problem.
4. 1. Definitions
Neuron coverage. We define neuron coverage of a set of test
inputs as the ratio of the number of unique activated neurons for all test inputs and the total number of neurons in
the DNN.d We consider a neuron activated if its output is
greater than a threshold (e.g., 0).
More formally, let us assume that all neurons of a DNN
are represented by the set N = {n1, n2, ...}, all test inputs are
represented by the set T = {x1, x2, ...}, and out(n, x) is a func-
tion that returns the output value of neuron n in the DNN
for a given test input x. Note that the bold x signifies that x
is a vector. Let t represent the threshold for considering a
neuron to be activated. In this setting, neuron coverage can
be defined as follows.
To demonstrate how neuron coverage is calculated in
practice, consider the DNN as shown in Figure 4b. The neuron coverage (with threshold 0) for the input picture of the
red car as shown in Figure 4b will be 5/8 = 0.625.
Gradient. The gradients or forward derivatives of the outputs of neurons of a DNN with respect to the input are well
known in deep learning literature. They have been extensively used both for crafting adversarial examples and visu-alizing/understanding DNNs.
18 We provide a brief definition
here for completeness and refer interested readers to18 for
more details.
Let q and x represent the parameters and the test input of
a DNN, respectively. The parametric function performed by
a neuron can be represented as y = f (q, x) where f is a func-
tion that takes q and x as input and output y. Note that y can
be the output of any neuron defined in the DNN (e.g., neu-
ron from output layer or intermediate layers). The gradient
of f (q, x) with respect to input x can be defined as:
( 1)
The computation inside f is essentially a sequence of
stacked functions that compute the input from previous
layers and forward the output to next layers. Thus, G can be
calculated by utilizing the chain rule in calculus, that is, by
computing the layer-wise derivatives starting from the layer
of the neuron that outputs y until reaching the input layer
that takes x as the input. Note that the dimension of the gradient G is identical to that of the input x.
4. 2. DeepXplore algorithm
The main advantage of the test input generation process for
a DNN over traditional software is that the test generation
process, once defined as an optimization problem, can be
solved efficiently using gradient ascent. In this section, we
describe the details of the formulation and find solutions to
the optimization problem. Note that solutions to the optimization problem can be efficiently found for DNNs as the gradients of the objective functions of DNNs, unlike traditional
software, can be easily computed.
As discussed earlier in Section 3, the objective of the
test generation process is to maximize both the number of
observed differential behaviors and the neuron coverage
while preserving domain-specific constraints provided by
the users. Below, we define the objectives of our joint optimization problem formally and explain the details of the
algorithm for solving it.
Maximizing differential behaviors. The first objective of
the optimization problem is to generate test inputs that can
induce different behaviors in the tested DNNs, that is, different DNNs will classify the same input into different classes.
Suppose we have n DNNs Fk∈
1..n: x → y, where Fk is the function modeled by the kth neural network. x represents the
input and y represents the output class probability vectors.
(a) DNNs produce same output (b) DNNs produce different output
NN1 NN2 NN1 NN2
Car
0.95
Car
0.98
Car
0.75
Face
0
Face
0
Face
0.24
Car
0.3
Face
0.69
... ... ... ...
Figure 6. Inputs inducing different behaviors in two similar DNNs.
d Neuron coverage can be defined in many different ways other than that
defined in this paper. We refer readers to other follow-up papers for details
on different definitions.
