Vviewpoints
• Peter G. Neumann, Column Editor
I
M
A
G
E
B
Y
A
N
D
R
I
J
B
O
R
Y
S
A
S
S
O
C
I
A
T
E
S
,
U
S
I
N
G
S
H
U
T
T
E
R
S
T
O
C
K
FROM WHAT I have seen, heard, and read, confusion and mis- information abound about software and safety. I have worked in this area for nearly
40 years, starting around the time when
computers were beginning to be introduced into the control of safety-critical
systems. I want to share what I have
learned. Too many incorrect beliefs are
being promoted, which are inhibiting
progress and, in some cases, unnecessarily costing lives. This column clari-fies this topic so that the solutions we
propose are more likely to have a significant impact on safety.
With only a few exceptions, software
was not used to directly control safety-critical systems until approximately
1980, although it was used to provide
computational power for complex systems, such as spacecraft. Direct control
was very limited, but the hesitation has
now almost completely disappeared
and software is used to control most
systems, including physical systems
that could involve potentially large and
even catastrophic losses.
Originally, “embedded software”
was used to denote these new control
roles for software, but more recently
the term “cyber-physical systems” has
come into vogue. The figure here shows
a standard cyber-physical control loop.
Note that, for some reason, cyber-phys-
ical systems usually forget that control
can be, and often is, provided by hu-
mans. In a little more realistic model
(but more complicated than necessary
here), there would be two controllers
where a human controller is providing
control signals to a computer control-
ler. To cover more than the unusual
case where there are no human con-
trollers, we should actually talk about
“cyber-human-physical” systems. Even
so-called “unmanned” air vehicles, for
example, usually have a human con-
troller on the ground. A more realistic
and complete model is provided in Ap-
pendix G of the S TPA Handbook. 4
As illustrated in the figure here, a
controller (or controllers, which may be
human, automated or both) compares
the current state of the controlled pro-
cess with the control goals and sends
control signals to an actuator, which in
turn may be automated or human. The
actuators translate the control signals
into physical actions on the controlled
process. Sensors provide feedback
about the state of the controlled pro-
cess to the controller so it can deter-
mine the state of the controlled system
and decide whether further control
signals are needed. The actuators and
sensors may be software, hardware,
physical devices, or humans.
In order to decide on what control
actions to provide in order to satisfy
its goals (requirements), the controller must have a model (often called
a mental model when the controller
is human) of the current state of the
controlled process. The most common
cause of accidents stemming from unsafe controller action is that the model
of the controlled process is incorrect:
the pilot thinks the aircraft is not in
a stall when it is and does not issue a
Inside Risks
Are You Sure Your Software
Will Not Kill Anyone?
Using software to control potentially unsafe systems requires
the use of new software and system engineering approaches.
DOI: 10.1145/3376127