Communications of the ACM

The sections that follow relate the small-batch principle to IT and show many benefits beyond reduced waste and improved latency.

In-House Software Deployment

A company had a team of developers that produced a new release every six months. When a release shipped, the operations team stopped everything and deployed the release into production. The process took three or four weeks and was very stressful for all involved. Scheduling the maintenance window required complex negotiation. Testing the release was complex and required all hands on deck. The actual software installation never worked on the first try. Once deployed, a number of high-priority bugs would be discovered, and each would be fixed by various “hot patches” that follow.

Even though the deployment process was labor intensive, there was no attempt to automate it. The team had many rationalizations that justified this.

The production infrastructure changed
significantly between releases, thus
making each release a moving target. It
was believed that any automation would
be useless by the next release because
each release’s installation instructions
were shockingly different. With each
next release being so far away, there
was always a more important “burning
issue” that had to be worked on first.

Thus, those that did want to automate the process were told to wait until tomorrow, and tomorrow never came. Lastly, everyone secretly hoped that maybe, just maybe, the next release cycle wouldn’t be so bad. Such optimism is a triumph of hope over experience.

Each release was a stressful, painful month for all involved. Soon it was known as hell month. To make matters worse, the new software was usually late. This made it impossible for the operations team to plan ahead. In particular, it was difficult to schedule any vacation time, which make everyone more stressed and unhappy.

Feeling compassion for the team’s woes, someone proposed that releases should be done less often, perhaps every 9 or 12 months. If something is painful, it is natural to want to do it less frequently.

To everyone’s surprise the operations team suggested going in the other direction: monthly releases.

This was a big-batch situation. To improve, the company didn’t need bigger batches, it needed smaller ones.

People were shocked! Were they proposing that every month be hell month?

No, by doing it more frequently, there
would be pressure to automate the pro-
cess. If something happens infrequent-
ly, there’s always an excuse to put off au-
tomating it. Also, there would be fewer
changes to the infrastructure between
releases. If an infrastructure change did
break the release automation, it would
be easier to fix the problem.
The change did not happen over-
night. First the developers changed
their methodology from mega releases
with many new features, to small it-
erations, each with a few specific new
features. This was a big change, and
selling the idea to the team and man-
agement was a long battle.

Meanwhile, the operations team automated the testing and deployment processes. The automation could take the latest code, test it, and deploy it into the beta-test area in less than an hour. The push to production was still manual, but by reusing code for the beta rollouts it became increasingly less manual over time.

The result was the beta area was updated multiple times a day. Since it was automated, there was little reason not to. This made the process continuous, instead of periodic. Each code change triggered the full testing suite, and problems were found in minutes rather than in months.