not a reasonable predictor for whether
that binary has a dependency on another service. If a standard mechanism
is used for specifying back ends or connection types—for example, if all back
ends are provided in configuration and
not in code—this might be an area
Restrictions are most effective if applied at runtime. By intercepting and
potentially blocking connections as
they are being made, you can be certain that you are inspecting the actual
behavior of the running system, rather
than speculating based on code or configuration. To avoid wasted engineering effort, restrictions on the back ends
that services may contact should be
implemented as early in the development life cycle as possible. Changing a
system’s architecture after it is already
live and in use is far more expensive. 6
By applying the same set of restrictions
at all stages of software development—
during development, testing, canarying, and running live—any unwelcome
dependency can be identified early.
There are several options for runtime enforcement. Just as with dependency tracking, existing infrastructure could be repurposed for
dependency control. If all interservice
connections pass through a firewall,
network device, load balancer, or service mesh, those infrastructure services could be instrumented to maintain a list of acceptable dependencies
and drop or deny any requests that
do not match the list. Silently dropping requests at a point between the
client and server may complicate
debugging, though. A request that
is dropped for being an unapproved
dependency may be indistinguishable from a failure of the server or the
intermediate device: the connections
may seem to just disappear.
Another option is to use a dedicated
external dependency-control service
that the client can query before allowing each new back-end connection.
This kind of external system has the
disadvantage of adding latency since
it requires extra requests to allow or
deny each back end. And, of course,
the dependency-control service itself
becomes a dependency of the service.
At Google, we had the most success
adding restrictions into the code at the
point where the connection is made.
production. Even so, the information
is available only after the new dependency has been created and is already
in use. This is sufficient for dependency tracking, where you want to describe
the interconnections of an existing system and become aware of new ones.
Preventing the dependency, however,
requires dependency control.
Dependency control. Just like
dependency tracking, dependency
control typically starts as a manual
process using information stored in
engineers’ heads. Developers might
include a list of proposed back ends
in all design documentation and depend on their colleagues’ knowledge
of the existing systems to flag dangers.
Again, this may be enough for a smaller environment. As services are born,
grow, change, and are deprecated, the
data can quickly become stale or unwieldy. Dependency control is most
effective if enforced programmatically, and there are several points to
consider in adding it.
When working on dependency management at Google, we found it best to
think about controlling dependencies
from the client side of a client-server
connection (that is, the service that is
about to depend on another service).
By owning the code that initiates the
connections, the owner of the client
has the most control and visibility over
which dependencies exist and can
therefore detect potential problems
earlier. The client is also most affected
by ill-considered dependencies.
Although the owner of a server may
want to control who its clients are for
reasons such as capacity planning or
security, bad dependencies are much
more likely to affect the client’s SLO.
Because the client requires some functionality or data from the server for its
own functionality or performance, it
needs to be prepared for server-side
outages. The server, on the other hand,
is unlikely to notice an outage of one of
One approach to dependency control is to analyze the client’s code and
restrict dependencies at build time.
The behavior of the binary, however,
will be influenced by the configuration
and environment it receives. Identical
binaries might have very different dependencies in different situations, and
the existence of code inside a binary is
control is to
analyze the client’s
code and restrict