More importantly, dashboards could be
configured and customized by individual users. They were self-service.
After much discussion, the team
decided to pivot to the other software
package. In the next iteration, they
set up the new software and created
an equivalent set of configurations.
This went very quickly because a lot
of work from the previous iterations
could be reused: the decisions on
what and how to monitor; the work
completed with the network team;
and so on.
By iteration 6, the entire team was
actively using the new software. Managers were setting up dashboards to display key metrics that were important to
them. People were enthusiastic about
the new system.
Something interesting happened
around this time: a major server
crashed on Saturday morning. The
monitoring system alerted the sysadmin team, who were able to fix the
problem before staff arrived at the office on Monday. In the past there had
been similar outages but repairs had
not begun until the sysadmins arrived
on Monday morning, well after most
employees had arrived. This showed
management, in a very tangible way,
the value of the system.
Iteration 7 had the goal of writing
a proposal to move the monitoring
system to physical machines so that
it would scale better. By this time the
managers who would approve such a
purchase were enthusiastically using
the system; many had become quite
expert at creating custom dashboards.
The case was made to move the system
to physical hardware for better scaling
and performance, and to use a duplicate set of hardware for a hot spare site
in another datacenter.
The plan was approved.
In future iterations the system became more valuable to the organization as the team implemented features such as a more sophisticated
on-call schedule, more monitored services, and so on. The benefits of small
batches observed by the sysadmin
˲ Testing assumptions early prevents
wasted effort. The ability to fail early
and often means the team can pivot.
Problems can be fixed sooner rather
˲ Providing value earlier builds momentum. People would rather have
some features today than all the features tomorrow. Some monitoring is
better than no monitoring. The naysay-ers see results and become advocates.
Management has an easier time approving something that isn’t hypothetical.
˲ Experimentation is easier. Often,
people develop emotional attachment
to code. With small batches they can be
more agile because they have grown less
attached to past decisions.
˲ Instant gratification. The team saw
the results of their work faster, which
˲ The team was less stressed. There is
no big, scary, due date, just a constant
flow of new features.
˲ Big-batch debating is procrastination. Much of the early debate had been
about details and features that didn’t
matter or didn’t get implemented.
The first few weeks were the hard-est. The initial configuration required
special skills. Once it was running,
however, people with less technical
skill or desire could add rules and
make dashboards. In other words, by
taking a lead and setting up the scaffolding, others can follow. This is an
important point of technical leadership. Technical leadership means going first and making it easy for others
A benefit of using the MVP model
is the system is always working. This
is called “always being in a shippable
state.” The system is always working
and providing benefit, even if not all
the features are delivered. Therefore,
if more urgent projects take the team
away, the system is still usable and running. If the original big-batch plan had
continued, the appearance of a more
urgent project might have left the system half developed and unlaunched.
The work done so far would have been
Why are small batches better?
Small batches result in happier customers. Features get delivered with less
latency. Bugs are fixed faster.
Small batches reduce risk. By testing assumptions, the prospect of future failure is reduced. More people
get experience with procedures, which
mean our skills improve.
Small batches reduce waste. They
avoid endless debates and perfectionism that delay the team in getting
started. Less time is spent implementing features that don’t get used. In
the event that higher-priority projects
come up, the team has already delivered a usable system.
Small batches improve the ability
to innovate. Because experimentation
is encouraged, the team can test new
ideas and keep the good ones. We can
take risks. We are less attached to old
pieces that must be thrown away.
Small batches improve productivity.
Bugs are fixed quicker and the process
of fixing them is accelerated because
the code is fresher in the mind.
Small batches encourage automation. When something must happen
often, excuses not to automate go away.
Small batches encourage experimentation. The team can try new
things—even crazy ideas, some of
which turn into competition-killing
features. We fear failure less because
we can easily undo a small batch if the
experiment fails. More importantly,
experimentation allows the team to
learn something that will help them
make future improvements.
Small batches make system administrators happier. We get instant gratification, and hell month disappears. It is
simply a better way to work.
Breaking The Major Release Habit
Adopting DevOps Practices in Quality
Thomas A. Limoncelli is a site reliability engineer at
Stack Overflow Inc. in NYC. His Everything Sysadmin
column appears in acmqueue ( http://queue.acm.org); he
blogs at EverythingSysadmin.com.
Reprinted with permission. Volume 1: The Practice of
System and Network Administration, 3rd Edition, by
Thomas A. Limoncelli, Christina J. Hogan, Strata R.
Chalup, Addison-Wesley Professional (Oct. 2016);
Copyright held by author.
Publication rights licensed to ACM. $15.00.