I
M
A
G
E
B
Y
I
R
I
N
A
V
I
N
N
I
K
O
V
A
flag unexpected code patterns. These
tools are a good first line of defense
against security vulnerabilities: they
are fast and can flag many shallow
bugs. Unfortunately, they are also
prone to report false alarms and they
do not catch every bug. Indeed, static
analysis tools are typically unsound
and incomplete in practice in order to
be fast and automatic.
Manual code inspection consists in
peer-reviewing code before releasing
it. It is part of most software-develop-
ment processes and can detect seri-
ous bugs. Penetration testing, or pen
testing for short, is a form of manual
code inspection where security experts
review code (as well as design and ar-
chitecture) with a specific focus on
security. Pen testing is flexible, appli-
cable to any software, easy to start (not
much tooling required), and can reveal
design flaws and coding errors that are
beyond the reach of automated tools.
But pen testing is labor-intensive, ex-
pensive, and does not scale well since
(good) pen testers are specialized and
in high demand.
Fuzzing is the third main approach
for hunting software security vulnera-
bilities. Fuzzing repeatedly executes an
application with all kinds of input vari-
ants with the goal of finding security
bugs, like buffer-overflows or crashes.
Fuzzing requires test automation, that
is, the ability to execute tests automati-
cally. It also requires each test to run
fast (typically in a few seconds at most)
and the application state to be reset af-
ter each iteration. Fuzzing is therefore
more difficult to set up when testing
complex distributed applications, like
cloud or server applications running
on multiple machines. In practice,
fuzzing is usually most effective when
applied to standalone applications
with large complex data parsers. For
each bug found, fuzzing provides one
or several concrete inputs that can be
used to reproduce and examine the
bug. Compared to static analysis, fuzz-
ing does not generate false alarms, but
it is more computationally expensive
(running for days or weeks) and it can
also miss bugs.

Over the last two decades, fuzzing has been shown to be remarkably ef-