that’s a problem,” but can’t show me
a bug ID. I say, “Bug ID, or it didn’t
Yes, the process is still being done
manually, but now each manual iteration is done by setting variables and
pasting lines of commands into the
terminal window. Each manual iteration tests the accuracy of the snippets
and finds new edge cases, bugs, and
better ways of verifying the results.
Phase 3: Create Automation
Soon these command-line snippets
are turned into longer scripts. Like
all good code, this is kept in a source
code repository. The artifacts begin
looking more like real software.
Perhaps the code is performing
only certain steps or works in only a
narrow set of circumstances. Each
manual iteration, however, expands
the code to cover new use cases. No
manual iteration should leave the
scripts unimproved. In fact, it should
be the other way around. Each manual
iteration is simply a test for the most
recent improvements. You should
look forward to finding an edge case
that breaks the code because this is
an opportunity to fix the problem.
Often the entire process is more
complex than is appropriate for a
scripting language. Turning snippets
of PowerShell or Bash into stand-alone scripts is easy, but it is often
better to write larger programs in languages such as Python, Ruby, or Go.
The individual snippets usually translate easily, and when they don’t, a reasonable stopgap measure is to have
the program “shell out” to run the
command line. These can be “
down-coded” into the native language later
Since February 2015, the SRE (site
A Culture of Automating
reliability engineering) team at Stack
Overflow has switched from a mix-
ture of Python and Bash to Go. Even
though Go isn’t a scripting language,
for small programs it compiles and
runs nearly as fast as Python takes to
start. At Stack Overflow we tend to pre-
fer compiled, type-checked languages
for large programs, especially when
multiple people are collaborating,
and, therefore, no one person is fa-
miliar with every line of code. Our pol-
icy was that Bash scripts couldn’t be
larger than 100 lines and Python pro-
manually only as a mechanism for
generating artifacts (documentation
and code snippets) that “move the
needle” toward an automated world.
People who are successful at automating tasks tend to work this way in every aspect of their jobs. It is just how
they work; it is part of their culture.
The successful engineer has a
quick way to create documents for
new procedures and to find existing
procedures. People with this mindset
avoid the cognitive load of deciding
whether or not a task is worth documenting, because they document everything. On finding a potential improvement, they are not slowed by the
speed bump of switching from a document viewer to a document editor
because they work from the editor at
the start. Heck, they have a dedicated
second monitor just for their editing
People with this culture revise documents in real time. Meanwhile, the
less successful engineer has a stack of
notes that he honestly plans on entering into a document someday soon—
perhaps the same “someday” when he
will start writing code.
The successful engineer realizes
that the earlier he starts collaborating, the sooner others can contribute.
Together they can create a culture of
documentation that spreads throughout the team. Thus, every project is
collaborative and has a “stone soup”
feeling, as all are invited to bring their
skills and insights. The more people
who embody this culture, the more
success it has.
This culture can be summarized
in two sentences: ( 1) Every manual
action must have a dual purpose of
completing a task and improving the
system. ( 2) Manual work should not
be tolerated unless it generates an artifact or improves an existing one.
Traditional software development involves requirements gathering and so
on. In the culture of automation, we
wiggle and iterate among four overlapping phases: document the steps;
create automation equivalents; create
automation; and create self-service
and autonomous systems.
Phase 1: Document the Steps
At the start, developers perform a task
manually to learn the process. They
keep good notes and record what they
do for each step. This is often exploratory or may require interviewing experts on how to do the process. They
produce an artifact—written documentation describing how the process is done.
Beginning programmers are
taught to write a program in pseudocode first, then turn each line of
pseudocode into actual code. The
same applies to automation: if you
can’t describe the process in writing,
you can’t automate it.
Documentation is automation.
Following a step-by-step guide is automation: you are the CPU; you are
following the instructions. As with
any prototyping language, you should
not expect perfection, but learning.
You have the benefit of being able to
spot and fix problems along the way.
You are a CPU that improves the code
as it executes!
There is no reason to wait for the
document to be perfect before moving on to the next phase. All that is required is that the people involved gain
the minimum necessary confidence
in the document to move forward.
Phase 2: Create
As the document matures, manual action generates a new kind of artifact:
command-line snippets. The document is augmented with automated
equivalents for each step.
At first, you simply paste the command line used to perform the step
into the document as is. The next
time you manually perform the task,
you improve it—perhaps by rewriting it. Over time these command examples become fully functional code
Other improvements happen.
Mouse clicks are replaced by commands. Quality assurance steps are
added then automated.
Mouse clicks and other GUI actions that have no API or command-line equivalent are noted. Bugs are
filed with the vendor and the bug ID
is added to the document. (As a manager, I am unsatisfied when engineers tell me, “Oh, the vendor knows