ory access from embedded microcon-trollers and input-output, and tampering.
As a reminder, some important mantras have been repeated in the Inside
Risks archives. Here are just a few:
˲Characterizing potential vulnerabilities inherent in various types of computer-related systems and operational
environments, such as automation, 11
clouds, 9 Io T, 5 and AI. 16
˲ System engineering and software
engineering as a discipline. 2, 15
˲ Theoretically based practice. 4, 14, 15
˲ Foresighted planning for achieving long-term benefits rather than just
short-term gains. 6, 8, 10
Note that old wisdom may still be
very relevant and insightful, as in the
Einstein quote, Norbert Wiener’s prescient Human Use of Human Beings, 18
and Don Norman’s The Design of Everyday Things. 13 Computer-Related Risks7 is
no exception. Furthermore, there is considerable hope in some recent advances.
For example, the CHERI hardware-soft-
ware architecture and its intra-process
its ongoing formal analysis of the hard-
ware specifications—can provide some
guidance on how many of the afore-
mentioned desiderata and principles12
can actually be constructively applied in
practice. CertiKos, seL4, and the Green
Hills separation kernel are other ex-
amples of formal analysis of real system
components—albeit just for operating-
Each bulleted item is oversimplified,
and the problems that must be faced
are complex and far-reaching. System
engineers, academics, computer users,
and others might wish to reflect on the
history of how we reached where we are
today, and how theoretical and practical
research and development experience
might help achieve the desired goals, as
well as avoiding known shortcomings
and as-yet-unrecognized vulnerabilities.
However, the bottom line is that we still
have a long way to go toward achieving
1. Abelson, H. et al. Keys Under Doormats: Mandating
Insecurity by Requiring Government Access to All
Data and Communications. July 6, 2015. https://
2. Bellovin, S. M. and Neumann, P. G. The big picture.
Commun. ACM 61, 11 (Nov. 2018).
3. Landwehr, C. J. et al. Software systems engineering
programmes: A capability approach. In Journal of
Systems and Software 125 (Mar. 2017), 354–364;
Article: JSS9898 doi 10.1016/j.jss.2016.12.016
4. Leveson, N. and Young, W. An integrated approach to
safety and security based on system theory. Commun.
ACM 57, 2 (Feb. 2014).
5. Lindqvist, U. and Neumann, P.G. Risks in the emerging
Internet of Things. Commun. ACM 55, 2 (Feb. 2017).
6. Neumann, P.G. The foresight saga, redux. Commun.
ACM 55, 10 (Oct. 2012).
7. Neumann, P.G. Computer-Related Risks.
Addison-Wesley and ACM Press, 1995.
8. Neumann, P.G. More sight on foresight. Commun. ACM
56, 2 (Feb. 2013).
9. Neumann, P.G. Risks and myths of cloud computing
and cloud storage. Commun. ACM 57, 10 (Oct. 2014).
10. Neumann, P.G. Far-sighted planning for deleterious
computer-related events. Commun. ACM 58, 2 (Feb. 2015).
11. Neumann, P.G. Risks of automation. Commun. ACM
59, 10 (Oct. 2016).
12. Neumann, P.G. Fundamental trustworthiness
principles in CHERI. A. Shrobe, D. Shrier, and A.
Pentland, Eds. In New Solutions for Cybersecurity,
MIT Press/Connection Science, 2018, chapter 6.
13. Norman, D. The Design of Everyday Things, 2002;
revised and expanded edition, 2013.
14. Parnas, D.L. Software engineering: An unconsummated
marriage. Commun. ACM 40, 9 (Sept. 1997).
15. Parnas, D. L. Risks of undisciplined development.
Commun. ACM 53, 10 (Oct. 2010).
16. Parnas, D.L. The real risks of artificial intelligence.
Commun. ACM 60, 10 (Oct. 2017).
17. Watson, R.N. M. Capability Hardware Enhanced RISC
Instructions: CHERI Instruction-Set Architecture,
Version 7, University of Cambridge, June 2019; https://
18. Wiener, N. The Human Use of Human Beings.
Houghton Mifflin, 1950, revised 1954.
Peter G. Neumann ( firstname.lastname@example.org) is Chief
Scientist of the SRI International Computer Science Lab,
and moderator of the ACM Risks Forum.
Copyright held by author.
Advertise with ACM!
Reach the innovators and thought leaders
working at the cutting edge of
computing and information
technology through ACM’s magazines,
websites and newsletters.
+ 1 212-626-0686
Request a media kit
with specifications and pricing:
Software Engineering Education
18 and 19
16 March 2020
Proceedings will be published in:
ECSEE2020_V02.indd 1 7/29/19 7:39AM