grams couldn’t be larger than 1,000 lines. Those seemed like reasonable limits. Rewriting scripts when they grew beyond the limit, however, was a lot of work. It was better to start in Go and avoid the conversion.

Phase 4: Self-Service
and Autonomous Systems

In the next phase the script becomes a stand-alone tool, which then becomes part of a larger system, usually with a Web-based front end. Ideally, some kind of self-service portal can be created so that users can activate the automation themselves. Even better is to create an autonomous system. The difference between automated and autonomous is the difference between a tool that someone can use to create new user accounts, and a system that monitors the HR database and creates and deletes accounts without human intervention. Autonomous systems eliminate the human task.

Depending on how frequently the task is needed, this phase may not be worth the effort. The return on investment may indicate that stopping at the tool stage is sufficient. CI (continuous integration) systems such as Jenkins and runbook automation systems such as Rundeck, however, make it easy to create simple, RBAC (role-based access control) restrained, self-service portals.

Discipline

Maintaining this culture and not backsliding takes discipline. Every manual iteration must move you closer to better automation.

It is tempting to revert to the old methods or skip updating the documentation “just this once” because you are in a hurry, or you’ll fix it next time, or the new system is broken, or you’re not in a good mood today. The developer mindset, however, resists such temptations and treats every manual iteration as an opportunity that should not be squandered.

Doing something manually “be-
cause it is faster” is often a sign that
engineers feel pressure, but they do
not realize they are mortgaging their
future. In reality, the old way may feel
faster only because they are more
comfortable with it. Often the time
pressure they feel does not actually
exist. Will the person who asked the
engineer to do this particular task
notice that it took 20 minutes in-
stead of five minutes? If the person
is in the middle of a two-hour meet-
ing, he or she certainly won’t notice.
A few extra minutes spent improving
the system, however, pays off in all
future iterations.
On the contrary, I have often de-
bugged brittle code in front of the re-
quester. I give the requester a choice:
Fix their problem quickly the manual
way, or allow me to take a little extra
time and fix the automation so that
all can benefit. We agree in advance
to a deadline at which point I’ll revert
to the reliable-but-manual process.
Technology and non-technical people
alike jump at the opportunity. I sus-
pect they do this because it feels good
to help fix a larger problem, but also
because it has entertainment value.
When it is tempting to revert to
the old way for expediency’s sake, it
is useful to remind yourself that the
benefit of automation is not always
speed. Automation that is slower but
less error-prone can be a net gain if
the errors take a long time to fix. Pre-
venting a single error that requires a
day of restoring data from backups
could be invaluable. Because I’m fat-
fingered and easily distracted, this is
a major motivation for me.
Another benefit is the consistency
that automation can bring. Increased
variation increases the cost of support
and makes other automation projects
more burdensome by increasing the
number of edge cases. For example,
at one site I discovered that half the
Linux systems used raw disk parti-
tions, while the others used Linux
LVM (Logical Volume Manager) to
manage disk storage. This complicat-
ed the monitoring system (which now
had to handle both variations), pro-
cedure documentation (which had to
be written and tested with both varia-
tions), and so on. Tasks that should
have taken minutes took hours (or
days) on the machines that could not
benefit from LVM’s flexibility. The
two variations did not exist for techni-
cal reasons. The installation process
was not automated, and the manual
process resulted in what I will politely
call “creativity,” where we would have
preferred conformity.
Manual work
should not
be tolerated
unless it generates
an artifact
or improves
an existing one.