Syntax checking and file-format
validation are often easy to add and
Object Notation) files, in particular,
have a syntax that is easy to check with
code but not with the eyeball. Without
basic automated checks you will see a
common pattern of PRs being submit-
ted, followed quickly by a PR with an
exasperated comment such as “Damn
JSON! Damn comma rule!”
These checks can be added at many
stages in the CI pipeline. Git has the
pre-commit checks that run on the sub-
mitter’s client and block Git commits
if they fail, but this can be disabled, or
they may fail entirely if they depend on
particular operating systems or servic-
es. I prefer configuring the PR system
to run tests instead. CI systems can be
configured to run tests on any new PRs
before they are sent to the team for ap-
proval. This assures the checks are not
skipped and can include more deep
testing than can be included in pre-
commit checks, which may be difficult
to write in an operating system-agnos-
tic manner. Pre-vetted PRs reduce the
amount of work for the IT team.
As more of the checklist is automated, the team can handle a larger
volume of PRs.
What I like about this evolution
is it points coding projects to where
they are needed the most. Rather
than trying to automate every test and
never shipping, you can ship early
and often, adding tests that solve real
problems as you learn which errors
are the most common.
By starting with many manual
checks and automating issues only
as demand directs your attention,
low-value work is discouraged. As
stated in books such as The Goal (by
E.M. Goldratt, 1984) and The Phoenix
Project (by G. Kim et al., 2013), it is
best only to expend effort directly at
the bottleneck: improvements upstream of the bottleneck simply create a bigger backlog; improvements
downstream of the bottleneck are
If all the validations, checks, and
testing can be automated, the system can automatically approve PRs.
While rare, the result is that users can
get their needs met 24/7 without delay, even if the IT team is asleep, sick,
or on vacation.
advocates have been extolling the benefits of using declarative languages for
configuration files for years.
What’s new is enabling people outside the IT team to submit PRs, the extensive use of automated testing, and
using a CI system to integrate all of this.
Earlier I described the experiences of a
user making a request. The experience
from the provider side is more interesting
because it evolves over time. In this example I’ll trace a somewhat fictionalized
evolution of the use of DNSControl.c This
is a domain specific language (DSL) that
describes domain name system (DNS)
configurations (zone data) and a compiler that processes it and then updates
various DNS service providers such as
AWS (Amazon Web Services) Route 53,
Google Cloud DNS, ISC BIND, and more.
The DNSControl configuration file
is stored in Git. A CI system is configured so that changes trigger the test-and-push cycle of DNSControl, and
updates to the file result in the changes
propagating to the DNS providers. This
is a typical IaC pattern. When changes
are needed, someone from the IT team
updates the file and commits it to Git.
To turn this into GitOps, just add
documentation. The team’s wiki is
updated to point people to the correct
Git repository. The file format is pretty
obvious: find the domain you want to
modify and add the record. The command dnscontrol check does some
basic checks. Send the PR and you’re
off to the races.
The PRs are reviewed by the IT
team. The review includes technical
concerns such as syntax and string
lengths, but also institutional poli-
cies such as ordering, capitalization,
and a ban on foul language. Initially
these checks are done manually. Once
the PR is approved, the CI system
takes over and triggers the test-and-
It is important that the manual
checks performed by the IT team are
enumerated in the user-facing documentation. This improves the likelihood that a PR is approved on the first
try (plus, it is rather unethical to enforce a rule that isn’t advertised). Having the checks enumerated in writing
also allows the IT team to be consistent in their policy enforcement. It
also serves as a training mechanism:
new IT team members can perform
the same as anyone else.
This is often good enough. The system meets the basic requirements: it is
self-service, work is distributed down
to the requestor, and automation does
the boring parts. All or most of the
checklist is performed manually; but
at a low rate of PRs, that is OK.
If the rate of PRs increases, it might
be time to evolve the system by automating some of the manual checks.
Maybe one is often forgotten or frequently done incorrectly. Maybe an
outage could have been prevented by
additional bounds checking, and new
input validation can be implemented.
In fact, after every deployment-related
outage, you should pause to ask if a
validation or other check could have
A declarative language describes a desired state rather than instructions of what to do.
For example, a desired state for a VM configuration system might state, “There should
be three virtual machines named foo1, foo2, and foo3.” When the file is processed the
first time, the three VMs are created. Processing the same configuration file a second
time will leave the system unchanged, as the machines already exist. The configuration
file is idempotent. If for some reason foo2 were deleted, processing the file again would
Contrast this with an imperative configuration language that states, “Add three new
virtual machines.” This would generate three additional machines every time it is run,
eventually exhausting all resources.
Processing a declarative configuration repeatedly brings a system to (or closer to)
a desired state. In fact, many configuration management systems run hourly to detect
and fix deviations.
IaC: Declarative Languages
for System Configuration