Vviewpoints

• Peter G. Neumann, Column Editor

I M A G E B Y A N D R I J B O R Y S A S S O C I A T E S , U S I N G S H U T T E R S T O C K

FROM WHAT I have seen, heard, and read, confusion and mis- information abound about software and safety. I have worked in this area for nearly 40 years, starting around the time when computers were beginning to be introduced into the control of safety-critical systems. I want to share what I have learned. Too many incorrect beliefs are being promoted, which are inhibiting progress and, in some cases, unnecessarily costing lives. This column clari-fies this topic so that the solutions we propose are more likely to have a significant impact on safety.

With only a few exceptions, software was not used to directly control safety-critical systems until approximately 1980, although it was used to provide computational power for complex systems, such as spacecraft. Direct control was very limited, but the hesitation has now almost completely disappeared and software is used to control most systems, including physical systems that could involve potentially large and even catastrophic losses.

Originally, “embedded software”
was used to denote these new control
roles for software, but more recently
the term “cyber-physical systems” has
come into vogue. The figure here shows
a standard cyber-physical control loop.
Note that, for some reason, cyber-phys-
ical systems usually forget that control
can be, and often is, provided by hu-
mans. In a little more realistic model
(but more complicated than necessary
here), there would be two controllers
where a human controller is providing
control signals to a computer control-
ler. To cover more than the unusual
case where there are no human con-
trollers, we should actually talk about
“cyber-human-physical” systems. Even
so-called “unmanned” air vehicles, for
example, usually have a human con-
troller on the ground. A more realistic
and complete model is provided in Ap-
pendix G of the S TPA Handbook. 4

As illustrated in the figure here, a
controller (or controllers, which may be
human, automated or both) compares
the current state of the controlled pro-
cess with the control goals and sends
control signals to an actuator, which in
turn may be automated or human. The
actuators translate the control signals
into physical actions on the controlled
process. Sensors provide feedback
about the state of the controlled pro-
cess to the controller so it can deter-
mine the state of the controlled system
and decide whether further control
signals are needed. The actuators and
sensors may be software, hardware,
physical devices, or humans.

In order to decide on what control actions to provide in order to satisfy its goals (requirements), the controller must have a model (often called a mental model when the controller is human) of the current state of the controlled process. The most common cause of accidents stemming from unsafe controller action is that the model of the controlled process is incorrect: the pilot thinks the aircraft is not in a stall when it is and does not issue a

Inside Risks
Are You Sure Your Software
Will Not Kill Anyone?
Using software to control potentially unsafe systems requires
the use of new software and system engineering approaches.
DOI: 10.1145/3376127