When new features are developed,
both new and old code paths commonly exist simultaneously, controlled
through the use of conditional flags.
This technique avoids the need for
a development branch and makes
it easy to turn on and off features
through configuration updates rather
than full binary releases. While some
additional complexity is incurred for
developers, the merge problems of
a development branch are avoided.
Flag flips make it much easier and
faster to switch users off new implementations that have problems. This
method is typically used in project-specific code, not common library
code, and eventually flags are retired
so old code can be deleted. Google
uses a similar approach for routing live traffic through different code
paths to perform experiments that can
be tuned in real time through configuration changes. Such A/B experiments
can measure everything from the performance characteristics of the code
to user engagement related to subtle
Google workflow. Several best practices and supporting systems are required to avoid constant breakage in
the trunk-based development model,
where thousands of engineers commit
thousands of changes to the repository
on a daily basis. For instance, Google
has an automated testing infrastructure that initiates a rebuild of all affected dependencies on almost every
change committed to the repository.
If a change creates widespread build
breakage, a system is in place to automatically undo the change. To reduce
the incidence of bad code being committed in the first place, the highly
customizable Google “presubmit” infrastructure provides automated testing and analysis of changes before
they are added to the codebase. A set of
global presubmit analyses are run for
all changes, and code owners can create custom analyses that run only on
directories within the codebase they
specify. A small set of very low-level
core libraries uses a mechanism similar to a development branch to enforce
additional testing before new versions
are exposed to client code.
An important aspect of Google cul-
ture that encourages code quality is the
expectation that all code is reviewed
a comment). Then, without leaving
the code browser, they can send their
changes out to the appropriate review-
ers with auto-commit enabled.
Piper can also be used without CitC.
Developers can instead store Piper
workspaces on their local machines.
Piper also has limited interoperability
with Git. Over 80% of Piper users today
use CitC, with adoption continuing to
grow due to the many benefits provided by CitC.
Piper and CitC make working productively with a single, monolithic
source repository possible at the scale
of the Google codebase. The design
and architecture of these systems were
both heavily influenced by the trunk-based development paradigm employed at Google, as described here.
Trunk-based development. Google
practices trunk-based development on
top of the Piper source repository. The
vast majority of Piper users work at the
“head,” or most recent, version of a
single copy of the code called “trunk”
or “mainline.” Changes are made to
the repository in a single, serial ordering. The combination of trunk-based
development with a central repository
defines the monolithic codebase model. Immediately after any commit, the
new code is visible to, and usable by,
all other developers. The fact that Piper
users work on a single consistent view
of the Google codebase is key for providing the advantages described later
in this article.
Trunk-based development is beneficial in part because it avoids the painful merges that often occur when it is
time to reconcile long-lived branches.
Development on branches is unusual
and not well supported at Google,
though branches are typically used
for releases. Release branches are cut
from a specific revision of the repository. Bug fixes and enhancements that
must be added to a release are typically
developed on mainline, then cherry-picked into the release branch (see
Figure 6). Due to the need to maintain
stability and limit churn on the release
branch, a release is typically a snapshot of head, with an optional small
number of cherry-picks pulled in from
head as needed. Use of long-lived
branches with parallel development
on the branch and mainline is exceedingly rare.
Piper and CitC
with a single,
at the scale of the