3. Bolton, R.J. and Hand, D.J. Statistical fraud detection:
A review. Statistical Science 17, 3 (2002), 235–249.
4. Carlini, N. and Wagner, D. Towards evaluating the
robustness of neural networks. arXiv preprint, 2016;
5. Dang, H., Yue, H., and Chang, E. C. Evading classifier
in the dark: Guiding unpredictable morphing using
binary-output blackboxes. arXiv preprint, 2017;
6. Glorot, X., Bordes, A., and Bengio, Y. Deep sparse
rectifier neural networks. In Proceedings of the 14th
International Conference on Artificial Intelligence
and Statistics (Ft. Lauderdale, FL, Apr. 11–13, 2011),
7. Goodfellow, I., Bengio, Y., and Courville, A. Deep
Learning. MIT Press, Cambridge, MA, 2016; http://
8. Goodfellow, I.J., Bulatov, Y., Ibarz, J., Arnoud, S.,
and Shet, V. Multi-digit number recognition from
Street View imagery using deep convolutional neural
networks. In Proceedings of the International
Conference on Learning Representations (Banff,
Canada, Apr. 14–16, 2014).
9. Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining
and harnessing adversarial examples. arXiv preprint,
10. Grosse, K., Papernot, N., Manoharan, P., Backes, M.,
and McDaniel, P. Adversarial perturbations against
deep neural networks for malware classification. In
Proceedings of the European Symposium on Research
in Computer Security (Oslo, Nor way, 2017).
11. Hinton, G., Vinyals, O., and Dean, J. Distilling the
knowledge in a neural network. arXiv preprint, 2015;
12. Huang, S., Papernot, N., Goodfellow, I., Duan, Y., and
Abbeel, P. Adversarial attacks on neural network
policies. arXiv preprint, 2017; https://arxiv.org/
13. Huang, A., Kwiatkowska, M., Wang, S., and Wu, M.
Safety verification of deep neural networks. In
Proceedings of the International Conference on
Computer-Aided Verification (2016); https://link.
14. Jarrett, K., Kavukcuoglu, K., Ranzato, M.A., and LeCun,
Y. What is the best multi-stage architecture for
object recognition? In Proceedings of the 12th IEEE
International Conference on Computer Vision (Kyoto,
Japan, Sept. 27–Oct. 4). IEEE Press, 2009.
15. Katz, G., Barrett, C., Dill, D., Julian, K., and
Kochenderfer, M. Reluplex: An efficient SMT solver
for verifying deep neural networks. In Proceedings
of the International Conference on Computer-Aided
Verification. Springer, Cham, 2017, 97–117.
16. Kurakin, A., Goodfellow, I., and Bengio, S. Adversarial
examples in the physical world. In Proceedings of the
International Conference on Learning Representations
17. Murphy, K.P. Machine Learning: A Probabilistic
Perspective. MI T Press, Cambridge, MA, 2012.
18. Nair, V. and Hinton, G.E. Rectified linear units improve
restricted Boltzmann machines. In Proceedings of the
International Conference on Machine Learning (Haifa,
Israel, June 21–24, 2010).
19. Papernot, N., Goodfellow, I., Sheatsley, R., Feinman,
R., and McDaniel, P. CleverHans v2.1.0: An adversarial
machine learning library; https://github.com/
20. Papernot, N., McDaniel, P., and Goodfellow, I.
Transferability in machine learning: From phenomena
to black-box attacks using adversarial samples. arXiv
preprint, 2016; https://arxiv.org/abs/1605.07277
21. Papernot, N., McDaniel, P., Goodfellow, I., Jha, S.,
Celik, Z.B., and Swami, A. Practical black-box attacks
against deep learning systems using adversarial
examples. In Proceedings of the ACM Asia Conference
on Computer and Communications Security (Abu
Dhabi, UAE). ACM Press, New York, 2017.
22. Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik,
Z.B., and Swami, A. The limitations of deep learning
in adversarial settings. In Proceedings of the 2016
IEEE European Symposium on Security and Privacy
(Saarbrücken, Germany, Mar. 21–24). IEEE Press,
23. Papernot, N., McDaniel, P., Sinha, A., and Wellman,
M. Towards the science of security and privacy
in machine learning. In Proceedings of the Third
IEEE European Symposium on Security and Privacy
(London, U.K.); https://arxiv.org/abs/1611.03814
24. Papernot, N., McDaniel, P., Wu, X., Jha, S., and
Swami, A. Distillation as a defense to adversarial
perturbations against deep neural net works. In
Proceedings of the 37th IEEE Symposium on Security
and Privacy (San Jose, CA, May 23–25). IEEE Press,
25. Russell, S. and Norvig, P. Artificial Intelligence: A
Modern Approach. Prentice-Hall, Englewood Cliffs,
NJ, 1995, 25–27.
26. Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L.,
Van Den Driessche, G., Schrittwieser, J., Antonoglou,
I., Panneershelvam, V., Lanctot, M. et al. Mastering
the game of Go with deep neural networks and tree
search. Nature 529, 7587 (2016), 484–489.
27. Stallkamp, J., Schlipsing, M., Salmen, J., and Igel,
C. Man vs. computer: Benchmarking machine
learning algorithms for traffic sign recognition.
Neural Networks (2012); https://doi.org/10.1016/j.
28. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S.,
Anguelov, D., Erhan, C., Vanhoucke, V., and Rabinovich,
A. Going deeper with convolutions. In Proceedings of
the IEEE Conference on Computer Vision and Pattern
Recognition. IEEE Press, 2015, 1–9.
29. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and
Wojna, Z. Rethinking the Inception architecture for
computer vision. ArXiv e-prints, Dec. 2015; https://
30. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J.,
Erhan, D., Goodfellow, I., and Fergus, R. Intriguing
properties of neural networks. In Proceedings
of the International Conference on Learning
31. Taigman, Y., Yang, M., Ranzato, M. A., and Wolf,
L. DeepFace: Closing the gap to human-level
performance in face verification. In Proceedings of the
Computer Vision and Pattern Recognition Conference.
IEEE Press, 2014.
32. Tramèr, F., Kurakin, A., Papernot, N., Boneh, D., and
McDaniel, P. Ensemble adversarial training: Attacks
and defenses. arXiv preprint, 2017; https://arxiv.org/
33. Tramèr, F., Zhang, F., Juels, A., Reiter, M. K., and
Ristenpart, T. Stealing machine learning models
via prediction APIs. In Proceedings of the USENIX
Security Conference (San Francisco, CA, Jan. 25–27).
USENIX Association, Berkeley, CA, 2016.
34. Wolpert, D.H. The lack of a priori distinctions between
learning algorithms. Neural Computation 8, 7 (1996),
35. Xu, W., Qi, Y., and Evans, D. Automatically evading
classifiers. In Proceedings of the 2016 Network and
Distributed Systems Symposium (San Diego, CA, Feb.
21–24). Internet Society, Reston, VA, 2016.
Ian Goodfellow ( email@example.com) is a staff
research scientist at Google Brain, Mountain View, CA,
USA, and inventor of Generative Adversarial Networks.
Patrick McDaniel ( firstname.lastname@example.org) is the
William L. Weiss Professor of Information and
Communications Technology in the School of Electrical
Engineering and Computer Science at Pennsylvania State
University, University Park, PA, USA, and a fellow of both
IEEE and ACM.
Nicolas Papernot ( email@example.com) is a Google
Ph.D. Fellow in Security in the Department of Computer
Science and Engineering at Penn State University,
University Park, PA, USA.
Copyright held by the authors.
particular problem; it may simply be
replaced by another equally vexing
category of vulnerabilities. The vastness of the set of all possible inputs
to a machine learning model seems
to be cause for pessimism. Even for
a relatively small binary vector, there
are far more possible input vectors
than there are atoms in the universe,
and it seems highly improbable that
a machine learning algorithm would
be able to process all of them acceptably. On the other hand, one may
hope that as classifiers become more
robust, it could become impractical
for an attacker to find input points
that are reliably misclassified by the
target model, particularly in the black-box setting.
These questions may be addressed
empirically, by actually playing out
the arms race as new attacks and new
countermeasures are developed. We
may also be able to address these questions theoretically, by proving the arms
race must converge to some asymptote. All these endeavors are difficult,
and we hope many will be inspired to
join the effort.
Author Nicolas Papernot is supported
by a Google Ph.D. Fellowship in Security. Research was supported in part
by the Army Research Laboratory under Cooperative Agreement Number
W911NF-13-2-0045 (ARL Cyber Security CRA) and the Army Research Office under grant W911NF-13-1-0421.
The views and conclusions contained in this article are those of the
authors and should not be interpreted as representing the official policies, either expressed or implied, of
the Army Research Laboratory or the
U.S. government. The U.S. government is authorized to reproduce and
distribute reprints for government
purposes notwithstanding any copyright notation hereon.
1. Arp, D., Spreitzenbarth, M., Hubner, M., Gascon, H.,
Rieck, K., and Siemens, C.E.R. T. Drebin: Effective and
explainable detection of Android malware in your
pocket. In Proceedings of the NDSS Symposium
(San Diego, CA, Feb.). Internet Society, Reston, VA,
2. Barreno, M., Nelson, B., Sears, R., Joseph, A.D.,
and Tygar, J.D. Can machine learning be secure?
In Proceedings of the 2006 ACM Symposium on
Information, Computer and Communications Security
( Taipei, Taiwan, Mar. 21–24). ACM Press, New York,
Watch the authors discuss
their work in this exclusive