thetically and acting compassionately.
Tools can and do help, but they can’t
make us care.
4. Good Fences Make
Boundary objects and abstractions give
needed structure, and containers make
good boundary objects, but they do not
eliminate the liminal space between the
metaphorical (or all-too-real) dev and
ops. When you implement microservices, how micro is micro? Even if you
have a well-defined service that does one
thing (somewhat) well, a good rubric is
whether the service’s health endpoint
can answer unambiguously. If the answer to “Is this working?” is “Wellllll...,”
that service isn’t micro enough.
Deciding what’s yours and what’s
theirs is the basis of every sibling-rivalry
détente. In Eric Brewer’s CAP theorem
you can pick two of consistency, availability, and partition tolerance as long
as one of them is partition tolerance,
because, as distributed systems expert
Caitie McCaffrey puts it, “physics and
math.” In a distributed system that contains humans in multiple time zones,
you’re inevitably going to have partitions, and waiting 10 hours for headquarters to wake up and make a decision is nobody’s idea of a good time. But
decentralized decision making means
distributing power to your human edge
nodes (sometimes a hard sell).
Empowering developer choice is facilitated by containers; there is always
a tension between what someone else
dictates and what you are convinced
you need. Making thoughtful decisions
about tools and architecture can help;
well-considered constraints can free us
from the decisions that are not bringing
us distinguishable benefit. Containers
can help define scope and reach of a
given tool or project, and deconstructing systems to human scale allows us to
comprehend their complexity.
Being able to reproduce a build al-
lows for separation of concerns. We
want this to be effective and yet not intro-
duce unnecessary barriers. The prover-
bial wall of confusion is all too real, built
on the tension between having incentive
to ship changes and being rewarded for
stability. Building just the right abstrac-
tions that empower independent teams
is worth taking the time to iterate on
(and, no, nobody gets it right immediate-
ly, because “right” will evolve over time).
We want to empower people with
as much agency as possible within the
constraints that work for our organizations. To determine the right constraints for you, you need to talk to your
teams. Think in terms of TCP instead
of UDP; you will need to SYN/ACK to
really understand what other humans
want. Nonviolent communication,
where you restate what you heard, is an
effective way to checksum your human
communications. (Bonus: techies will
appreciate this logic!)
5. Avoiding Sadness as a Service
Hindsight being what it is, we can
look back and recognize inflection
points. It’s more difficult to recognize
change in the moment, but the days
of operating your own data centers,
where your unit of currency is the virtual machine, are coming to a definite
middle. The hipsters among us will
say that’s over and sell you on serverless (which is just servers you cannot
ssh into), but we are talking about the
realities of enterprise adoption here,
and they are about at the point of taking containers seriously. Application
container clustering is better for utilization and flexibility of workload
placement, and using containerized
abstractions makes for better portability, including for those orgs looking toward public cloud.
W. Edwards Deming, a leader in
the field of quality control, said, “It’s
not necessary to change. Survival is
not mandatory.” Change is difficult.
Not changing is even worse. Tools are
essential, but how we implement the
tools and grow the culture and practices in our organizations needs even
more attention. As it turns out, it’s not
mandatory to write a Markov bot to
parse the front page of Hacker News,
then yolo absolutely everything out to
Whether you are just starting to implement technical and organizational
change, or facing the prospect that you
already have legacy microservices, it’s
worth considering the why and how of
our behaviors, not just the what. If legacy were not important, you could just
turn it off. But this is where your customers and money live. Glorifying exciting
greenfield projects is all well and good,
but the reality is that bimodal IT is a lie.
It’s ludicrous to tell people that some of
them have to stay in “sad mode” indefinitely, while others catapult ahead in
“awesome mode.” Change is on a continuum; absolutely every change ever
doesn’t happen at the same instant.
We succeed when we share responsibility and have agency, when we move
past learned helplessness to active listening. Don’t be a named pipe; you are
not keyboard-as-a-service. Assuming we
can all read code, putting detail in your
commit messages can be a lot more
useful than soon-to-be-outdated comments. Tell future-you why you did
that thing; they can read but don’t
know what you intended. Oral tradition is like never writing state to disk;
flush those buffers. There is no flow-chart, no checklist, no shopping list of
ticky boxes that will make everything
better. “Anyone who says differently
is selling something,” as The Princess
Bride teaches us. Orgs have “the way
we do things” because process is the
scar tissue of past failures.
You cannot take delivery of a shipping container with 800 units of DevOps, and have 600 of them go to the people in awesome mode, while the people
in sad mode can look at the other 200
but not touch them. DevOps is something you do, not something a vendor
implements for you with today’s shiniest tools. Change for the better is a decision we make together.
Tools are necessary but not sufficient. To build a future we all can live
with, we have to build it together.
Editor’s Note: To read this article complete with
embedded hyperlinks, visit https://queue.acm.org/detail.
The Verification of a Distributed System
Adopting DevOps Practices
in Quality Assurance
Bridget Kromhout is a principal cloud developer
advocate at Microsoft. She leads the devopsdays
organization globally and the DevOps community at
home in Minneapolis, MN, USA. She podcasts with
Arrested DevOps, blogs at bridgetkromhout.com, and
tweets at twitter.com/bridgetkromhout.
Copyright held by owner/author.
Publication rights licensed to ACM. $15.00.