The sections that follow relate the
small-batch principle to IT and show
many benefits beyond reduced waste
and improved latency.
In-House Software Deployment
A company had a team of developers
that produced a new release every six
months. When a release shipped, the
operations team stopped everything
and deployed the release into production. The process took three or four
weeks and was very stressful for all involved. Scheduling the maintenance
window required complex negotiation.
Testing the release was complex and
required all hands on deck. The actual
software installation never worked on
the first try. Once deployed, a number
of high-priority bugs would be discovered, and each would be fixed by various “hot patches” that follow.
Even though the deployment process was labor intensive, there was no
attempt to automate it. The team had
many rationalizations that justified this.
The production infrastructure changed
significantly between releases, thus
making each release a moving target. It
was believed that any automation would
be useless by the next release because
each release’s installation instructions
were shockingly different. With each
next release being so far away, there
was always a more important “burning
issue” that had to be worked on first.
Thus, those that did want to automate
the process were told to wait until tomorrow, and tomorrow never came.
Lastly, everyone secretly hoped that
maybe, just maybe, the next release cycle wouldn’t be so bad. Such optimism
is a triumph of hope over experience.
Each release was a stressful, painful month for all involved. Soon it was
known as hell month. To make matters
worse, the new software was usually
late. This made it impossible for the operations team to plan ahead. In particular, it was difficult to schedule any vacation time, which make everyone more
stressed and unhappy.
Feeling compassion for the team’s
woes, someone proposed that releases
should be done less often, perhaps
every 9 or 12 months. If something is
painful, it is natural to want to do it
less frequently.
To everyone’s surprise the operations team suggested going in the other direction: monthly releases.
This was a big-batch situation. To
improve, the company didn’t need bigger batches, it needed smaller ones.
People were shocked! Were they proposing that every month be hell month?
No, by doing it more frequently, there
would be pressure to automate the pro-
cess. If something happens infrequent-
ly, there’s always an excuse to put off au-
tomating it. Also, there would be fewer
changes to the infrastructure between
releases. If an infrastructure change did
break the release automation, it would
be easier to fix the problem.
The change did not happen over-
night. First the developers changed
their methodology from mega releases
with many new features, to small it-
erations, each with a few specific new
features. This was a big change, and
selling the idea to the team and man-
agement was a long battle.
Meanwhile, the operations team
automated the testing and deployment processes. The automation
could take the latest code, test it, and
deploy it into the beta-test area in less
than an hour. The push to production
was still manual, but by reusing code
for the beta rollouts it became increasingly less manual over time.
The result was the beta area was updated multiple times a day. Since it was
automated, there was little reason not
to. This made the process continuous,
instead of periodic. Each code change
triggered the full testing suite, and
problems were found in minutes rather than in months.
Pushes to the production area hap-
pened monthly because they required I M
A
G
E
B
Y
E
L
E
N
A
E
L
I
S
S
E
E
V
A