code errors. Details on how these problems were found were also included: review, analysis (manual inspection used
to reverse engineer artifacts), observation (manual inspection not related to
the specific artifact being worked on),
beta test/functional test, structural coverage analysis, or system test.
Of all the problems found, 77%
were discovered by engineers using
engineering judgment, including 54%
found by analysis, 8% by observation,
and 15% by review. That is, three-quarters of problems with the software were
found by engineers looking hard to determine if the software is doing what it
should do. A successful verification effort can be done only on a set of project
artifacts that has been through a successful validation effort.
software Verification
Software verification can be accomplished by any of several means or
a combination. The most common
means is by test. Requirements-based
testing uses a set of requirements as
the basis for the test criteria and produces results that conclusively report
whether the software under test fulfills
those requirements.
A second common verification
method is analysis. Certain requirements are difficult or impossible to
verify by test. For example, if a requirement demands that interrupts be
locked and a critical section formed
during a particular operation, then interrupt locking may be impossible to
detect from the outside. In this case, it
is appropriate to produce a small analysis document that provides evidence
that this requirement is fulfilled.
sample traceability.
Test and analysis are by no means
the only verification options available.
For example, formal methods can be
used to prove correctness of an algorithm or fidelity to the software requirements. For small systems such as a
pressure sensor, exhaustive input
testing can be used to fully cover the input
space for the software. Even product
service history can be used in certain
limited situations.
structural coverage analysis
DO-178B says the following about SCA
(structural coverage analysis): “The ob-
jective of this analysis is to determine
which code structure was not exercised
by the requirements-based test proce-
dures…The structural coverage analysis
may be performed on the Source Code,
unless the software level is A and the
compiler generates object code that is
not directly traceable to Source Code
statements. Then, additional verifica-
tion should be performed on the ob-
ject code to establish the correctness
of such generated code sequences. A
compiler-generated array-bound check
in the object code is an example of ob-
ject code that is not directly traceable to
the Source Code.”
The idea behind SCA is simple: if
testing is complete and the software
fulfills the requirements completely,
then one would expect 100% of the
code would be executed during the test.
There are reasons why this might not
be true, however. For example, robust-
ness checks in the software might have
execution paths that are unreachable
because it is impossible under normal
testing conditions to create the excep-
tions they guard against.
Gaps in the SCA might also indicate
other problems: shortcomings in the
requirements-based test cases or procedures, inadequacies in the software requirements, or dead code, which is code
that cannot be traced to any requirements. Dead code should be removed
from the system or the requirements
should be updated if that function is
necessary for the software.
Deactivated code is not intended to
be executed during flight, although it
has corresponding requirements and
should be present in the software. (
Perhaps this code is used only for ground
maintenance activities.) It is necessary
to show that deactivated code cannot be
inadvertently executed in a mode where
the code is not intended to run.
In addition to the activities showing
coverage of source or object code in the
software under test, further analysis
must be performed to confirm the data
coupling and control coupling between
the code components. The output of
the compiler is not trusted. Testing and
structural coverage analysis is used to
confirm that source code and object
code are fully covered and any discrep-ancies are remedied or documented.
Similarly, the linker (or any other reference fix-up mechanism) is also not
trusted. Proper coupling and fix-up
must be shown to be correct. SCA data
along with control coupling data are reviewable artifacts.
The accompanying figure shows the
web of traceability from HLR (high-level
requirement) to SCA report and its review checklist. Dashed lines indicate
traceability. Note that this diagram is
only representative (and not complete).
For example, documents used for verification by analysis are not shown. Even
so, this gives a good general overview of
a typical web of traceability.
review Checklist review Checklist
review Checklist
Design
Code
sCA Data
sCA report
hLr
LLr
Tests
Test results
review Checklists review Checklists review Checklist review Checklist
Traceability
Development and Verification tools
As previously mentioned, neither the
compiler nor the linker is trusted fully.
These are considered development tools
because they create artifacts that fly. It
is possible to create qualified development tools whose output need not be
checked. The guidance for the creation
of such a tool (and its verification) is
the same as the guidance for the creation and verification for software that
flies. At times, however, the extraordinary burden of creating that full set of
