As a result, an incremental and
iterative approach is required. Early
phases are more exploratory, and
later phases are more confident. You
start by working on the low-hanging
fruit, not because they are easy, but
because if you are honest with yourself, you have to admit to having no
idea how the more difficult parts
could ever conceivably be implemented. By doing the easier parts, however,
you gain the experience that makes
the other parts possible. Initial experiences inform later decisions, build
confidence, and give you the fortitude
to continue. Soon the impossible
parts of the project become possible.
Therefore, working in a waterfall
approach is untenable. Maintaining a
lockstep workflow through the phases would mean never leaving the first
gate. Some steps may be ready for full
automation, while others lag behind.
You cannot wait for the documentation to be perfect before moving to the
next phase. You may not have figured
out a command-line equivalent for
step 46, but the other steps can move
forward. I once used a system that
was pretty darn automated, except
someone had to be there to click “ok”
at one point. It took months to eliminate that. I’m glad we didn’t wait.
Enable Early Collaboration
An iterative structure improves your
ability to work collaboratively. If the
documentation is on a wiki or similar
system, everyone can contribute and
update the documentation. Once the
basic infrastructure is in place, everyone can fill in the missing pieces by
adding support for new edge cases,
improving testing, and so on. Good
engineers build the initial framework
but make it easy for others to contribute. I call this the “stone soup”
method of software development: you
bring the cooking pot and everyone
else fills it.
The earlier you share, the better.
The earlier you can enable this collaboration, the sooner more people can
contribute. For example, by keeping
the documentation in something easy
to edit, such as a wiki or Git repository, everyone on the team can “be the
CPU,” not only testing the algorithm,
but also contributing improvements.
The sooner the software is packaged
in a way that everyone can use, the
sooner feedback is available. Some-
one with a developer mindset treats
the documentation and code a lot like
an open source project—available
and easy to contribute to.
To enable collaboration, use the
same tools people are already using. If your team uses Git, keep the
documentation in Git. Repurpose the
team’s wiki, Google docs structure, CI
system, or whatever will lower the bar
The anti-pattern is to work privately and plan on releasing the documentation and code to the rest of the
team “next week.” Next week never
comes. It is a red flag when I hear
someone say “the code isn’t ready to
share with other people” or “I can’t
show the document to the team until
the next round of edits.” The opposite
is true. If you release something you
think “works only for you,” it enables
others to figure out how to make it
run for them. How can you know what
parts work only for you if you haven’t
let other people try it?
It is important for managers to
create a structure where projects are
easily sharable from the start, and
to provide (gentle) pressure to move
projects into that structure when
they are not. I try to role-model the
release-early attitude by starting my
documentation and code in an open
Git repository, unabashedly inserting comments such as “This code
sucks and needs to be replaced,” or
by indicating which parts are missing or could use improvement. Do not
shame people for releasing broken
code; reward them for transparency
and promoting collaboration.
Some IT engineers never have time to
automate their work. Others have the
same time constraints but succeed
in creating the preconditions (
documentation, code snippets) that enable automation.
As you work, you have a choice. Will
each manual task create artifacts that
allow you to accelerate future work, or
do you squander these opportunities
and accept the status quo?
By constantly documenting and
creating code-snippet artifacts, you
accelerate future work. That one-shot
task that could never happen again,
does happen again, and next time it
moves faster. Even tasks that aren’t
worth automating can be improved by
documenting them, as documenta-
tion is automation.
Every IT team should have a culture of constant improvement—or
movement along the path toward the
goal of automating whatever the team
feels confident in automating, in
ways that are easy to change as conditions change. As the needle moves to
the right, the team learns from each
other’s experiences, and the system
becomes easier to create and safer to
A good team has a structure in
place that makes the process frictionless and collaborative—plus, management that rewards and encourages the developer’s mindset. Always be
This article benefited from feedback
from John Allspaw (Adaptive Capacity Labs), Nicole Forsgren (DORA:
DevOps Research and Assessment
LLC), and Jason Shantz (Stack Overflow, Inc.).
The Small Batches Principle
Thomas A. Limoncelli
Swamped by Automation
Automated QA Testing
at EA: Driven by Events
1. Allspaw, J. A mature role for automation, part
II. Kitchen Soap; https://www.kitchensoap.
2. Bainbridge, L. Ironies of automation. Automatica 19, 6
4. Limoncelli, T.A. Automation should be like Iron Man,
not Ultron. acmqueue 13, 8 (2015); https://queue.acm.
Thomas A. Limoncelli is the site reliability engineering
manager at Stack Overflow Inc. in New York City. His
books include The Practice of System and Network
Administration, The Practice of Cloud System
Administration, and Time Management for System
Administrators. He blogs at EverythingSysadmin.com and
tweets Yes That Tom.
Copyright held by owner/author.
Publication rights licensed to ACM. $15.00.