31. Peisert, S., et al. ASCR Cybersecurity for Scientific
Computing Integrity. TR LBNL-6953E, U. S.
Department of Energy Office of Science, Feb. 2015.
32. Peisert, S. et al. ASCR Cybersecurity for Scientific
Computing Integrity|Research Pathways and Ideas
Workshop. TR LBNL-191105, U. S. Department of
Energy Office of Science, Sept. 2015.
33. Pérez, F. and Granger, B.E. IPython: A System for
interactive scientific computing. Computing in Science
and Engineering 9, 3 (May 2007), 21–29.
34. Popa, R.A., Redfield, C., Zeldovich, N. and Balakrishnan,
H. Cryptdb: Processing queries on an encrypted
database. Commun. ACM 55, 9 (Sept. 2012), 103–111.
35. Popa, R.A., Stark, E., Helfer, J., Valdez, S., Zeldovich,
N., Kaashoek, M.F. and Balakrishnan, H. Building Web
applications on top of encrypted data using Mylar.
In Proceedings of the 11th Symposium on Networked
Systems Design and Implementation (2014), 157–172.
36. Rubio-Gonzàlez, C. Precimonious: Tuning assistant
for floating-point precision. In Proceedings of the
International Conf. on High Performance Computing,
Networking, Storage and Analysis. ACM, 2013, 27.
37. Reubel, O. WarpIV: In situ visualization and analysis of
ion accelerator simulations. IEEE Computer Graphics
and Applications 36, 3 (2016), 22–35.
38. Ramakrishnan, L., Poon, S., Hendrix, V., Gunter, D.,
Pastorello, G.Z. and Agarwal, D. Experiences with
user-centered design for the Tigres workflow API.
In Proceedings of 2014 IEEE 10th International
Conference on e-Science, vol 1. IEEE, 290–297.
39. Singer A. Tempting fate. ;login: 30, 1 (Feb. 2005), 27–30.
40. Schneier, B. and Kelsey, J. Automatic event-stream
notarization using digital signatures. In Proceedings of
the 4th International Workshop on Security Protocols.
Springer, 1996, 155–169.
41. Sommer, R. and Paxson, V. Outside the closed world:
On using machine learning for network intrusion
detection. In Proceedings of the 31st IEEE Symposium
on Security and Privacy, Oakland, CA, May 2010.
42. Stoll, C. Stalking the wily hacker. Commun. ACM 31, 5
(May 1988), 484–497.
43. Skinner, D., Wright, N., Fürlinger, K., Yelick, K.A. and
Snavely, A. Integrated Performance Monitoring;
44. Wallace, D. Compute node Linux: New frontiers in compute
node operating systems. Cray User Group, 2007.
45. Whitlock, B., Favre, J.M. and Meredith, J.S. Parallel
in situ coupling of simulation with a fully featured
visualization system. In Proceedings of the 11th
Eurographics Conference on Parallel Graphics and
Visualization, 2011, 101–109.
46. Wisniewski, R. W., Inglett, T., Keppel, P., Murty, R.
and Riesen, R. mOS: An architecture for extreme-scale operating systems. In Proceedings of the 4th
International Workshop on Runtime and Operating
Systems for Supercomputers. ACM, 2014.
47. Whalen, S., Peisert, S. and Bishop, M. Network-theoretic
classification of parallel computation patterns. In
Proceedings of the First International Workshop on
Characterizing Applications for Heterogeneous Exascale
Systems (Tucson, AZ, June 4, 2011).
48. Whalen, S., Peisert, S. and Bishop, M. Multiclass
Classification of Distributed Memory Parallel
Computations. Pattern Recognition Letters 34, 3 (Feb.
49. Yosinski, J., Clune, J., Fuchs, T. and Lipson, H.
Understanding neural networks through deep
visualization. In Proceedings of the Deep Learning
Workshop, International Conference on Machine
50. Yelick, K. A Superfacility for Data Intensive Science.
Advanced Scientific Computing Research Advisory
Committee, Washington, DC, Nov. 8, 2016; http://science.
Sean Peisert ( firstname.lastname@example.org) is Staff Scientist
at Lawrence Berkeley National Laboratory, Chief
Cybersecurity Strategist at CENIC, and an associate
adjunct professor at the University of California, Davis.
Copyright held by owner/author.
but they also have some significant differences. This article presented both
challenges and opportunities.
Two key security challenges are the
notions that traditional security solutions often are not effective given the
paramount priority of high-performance in HPC. In addition, the need
to make some HPC environments as
open as possible to enable broad scientific collaboration and interactive HPC
also presents a challenge.
There may also be opportunities, as
described by the four themes regarding HPC security presented here. The
fact that HPC systems tend to be used
for very distinctive purposes, notably mathematical computations, may
mean the regularity of activity within
HPC systems can benefit the effectiveness of machine learning analyses
on security monitoring data to detect
misuse of cycles and threats to computational integrity. In addition, custom stacks provide opportunities for
enhanced security monitoring, and
the general trend toward containerized operation, limited interfaces, and
reduced complexity in HPC is likely
to help in the future much as reduced
complexity has benefitted the Science
Appreciation to Deb Agarwal, David
Brown, Jonathan Carter, Phil Colella,
Dan Gunter, Inder Monga, and Kathy
Yelick for their valuable feedback and
to Sean Whalen and Bogdan Copos
for their excellent work underlying the
ideas for new approaches described
here. Thanks to Glenn Lockwood for
his insights on the specifications for
the DOE ASCR hardware and software
coming in the next few years, and both
Glenn Lockwood and Scott Campbell
for the time spent providing the data
that supported that research.
This work used resources of the National Energy Research Scientific Computing Center and was supported by the
Director, Office of Science, Office of Advanced Scientific Computing Research,
of the U.S. Department of Energy under
Contract No. DE-AC02-05CH11231.
Any opinions, findings, conclusions,
or recommendations expressed in this
material are those of the author and do
not necessarily reflect those of the employers or sponsors of this work.
1. Adiga, N.R. et al. An overview of the Blue-Gene/L
supercomputer. In Proceedings of the ACM/IEEE
Conference on Supercomputing, 2002.
2. Austin, B. et al. 2014 NERSC Workload Analysis (Nov.
5., 2015); http://portal.nersc.gov/project/mpccc/
3. Anderson, R.J. UEPS: A second-generation electronic
wallet. In Proceedings of the 2nd European Symposium
on Research in Computer Security (Nov. 1992), 411–418.
4. Bailey, D.H. Resolving numerical anomalies in scientific
5. Bailey, D. H., Borwein, J.M. and Stodden, V. Facilitating
reproducibility in scientific computing: Principles
and practice. Reproducibility: Principles, Problems,
Practices. H. Atmanspacher and S. Maasen, Eds. John
Wiley and Sons, New York, NY, 2015.
6. Bailey, D.H., Demmel, J., Kahan, W., Revy, G. and Sen,
K. Techniques for the automatic debugging of scientific
floating-point programs. In Proceedings of the 14th
GAMM-IMACS International Symposium on Scientific
Computing, Computer Arithmetic and Validated
Numerics (Lyon, France, Sept. 2010).
7. Bishop, M. Computer Security: Art and Science.
Addison-Wesley Professional, Boston, MA, 2003.
8. Cappello, F. Improving the trust in results of numerical
simulations and scientific data analytics. 2015.
9. CoreOS, Inc. rkt - App Container runtime. https://
10. Cray, Inc. Cray Linux Environment Software Release
Overview, s-2425-52xx edition (Apr 2014); http://docs.
11. DARPA. Transparent Computing; http://www.
12. Das, A., Agrawal, H., Zitnick, C.L., Parikh, D. and Batra,
D. Human attention in visual question answering: Do
humans and deep networks look at the same regions?
In Proceedings of the Conference on Empirical
Methods in Natural Language Processing, 2016.
13. Dart, E., Rotman, L., Tierney, B., Hester, M. and Zurawski,
J. The science DMZ: A network design pattern for
data-intensive science. In Proceedings of the IEEE/ACM
Annual Super Computing Conference (Denver CO, 2013).
14. DeMasi, O., Samak, T. and Bailey, D. H. Identifying HPC
codes via performance logs and machine learning. In
Proceedings of the Workshop on Changing Landscapes
in HPC Security (2013).
15. Dwork, C. Differential privacy. In Proceedings of the
33rd International Colloquium on Automata, Languages
and Programming, Part II. Lecture Notes in Computer
Science 4052, (July 2006), 1–12. Springer Verlag.
16. Gefter, A. Is artificial intelligence permanently
inscrutable? Nautilus 40 (Sept. 1, 2016).
17. Gentry, C. Computing arbitrary functions of encrypted
data. Commun. ACM 53, 3 (Mar. 2010), 97–105.
18. Haber, S. and Stornetta, W.S. How to time-stamp a
digital document. J. Cryptology 3, 2 (Jan. 1991), 99–111.
19. Jacobsen, D.M. and Canon, R.S. Contain this,
unleashing docker for HPC. Proceedings of the Cray
User Group, 2015.
20. Jiang, L. and Su, Z. Osprey: A practical type system for
validating dimensional unit correctness of c programs.
In Proceedings of the 28th International Conference on
Soft ware Engineering, (2006), 262–271 ACM, New York.
21. KBase: The Department of Energy Systems Biology
22. Kasiviswanathan, S. P., Lee, H.K., Nissim, K.,
Raskhodnikova, S. and Smith, A. What can we learn
privately? SIAM J. Computing 40, 3 (2011), 793–826.
23. Kurtzer, G. M. et al. Singularity; http://singularity.lbl.gov.
24. Marko, J. and Bergman, L. Internet attack is called
broad and long lasting. New York Times (May 10, 2005).
25. Merkel, D. Docker: Lightweight Linux containers for
consistent development and deployment. Linux J. 239
26. Nakamoto, S. Bitcoin: A Peer-to-Peer Electronic Cash
System (May 24, 2009); http://www.bitcoin.org/bitcoin.pdf.
27. Nataraj, A., Malony, A.D., Morris, A. and Shende, S.
Early experiences with KTAU on the IBM BG/L.
In European Conference on Parallel Processing, pp.
99-110. Springer, 2006.
28. Paxson, V. Bro: A system for detecting network
intruders in real time. Computer Networks 31, 23
29. Peisert, S., et al. The Medical Science DMZ. J. American
Medical Informatics Assoc. 23, 6 (Nov. 1, 2016).
30. Peisert S. Fingerprinting Communication and
Computation on HPC Machines. TR LBNL-3483E,
Lawrence Berkeley National Laboratory, June 2010.
Watch the author discuss
his work in this exclusive