˲ reduce the amount of time spent
A key element in achieving these
goals was implementing continuous
delivery, with a particular focus on:
˲ the practice of continuous integration.
˲ significant investment in test automation.
˲ creation of a hardware simulator
so that tests could be run on a virtual
˲ reproduction of test failures on developer workstations.
After three years of work, the HP
LaserJet Firmware division changed
the economics of the software delivery
process by adopting continuous delivery, comprehensive test automation,
an iterative and adaptive approach to
program management, and a more
agile planning process. The economic
benefits were substantial:
˲ Overall development costs were reduced by approximately 40%.
˲ Programs under development increased by approximately 140%.
˲Development costs per program
went down 78%.
˲ Resources driving innovation increased eightfold.
For more on this case study, see
Leading the Transformation: Applying
Agile and DevOps Principles at Scale by
Gary Gruver and Tommy Mouser.
The most important point to remember from this case study is that the enormous cost savings and improvements
in productivity were possible only with
a large and ongoing investment by the
team in test automation and continuous integration. Even today, many people think that lean is a management-led
activity and that it’s about simply cutting costs. In reality, it requires investing to remove waste and reduce failure
demand—it is a worker-led activity that
can continuously drive down costs and
improve quality and productivity.
Handling legacy systems. Many
organizations hold mission-critical
data in systems designed decades
ago, often referred to as legacy systems. The principles and practices
of continuous delivery, however, can
be applied effectively in the context
of mainframe systems. Scott Buckley
and John Kordyback describe how
Suncorp, Australia’s biggest insurance
company, did exactly this.
(CDE) from the rest of the system, limiting the scope of the PCI DSS regulations to one segregated area and preventing them from ‘leaking’ through
to all their production systems. The
systems that form the CDE are separated (and managed differently) from
the rest of Etsy’s environments at the
physical, network, source code, and
logical infrastructure levels. Furthermore, the CDE is built and operated by
a cross-functional team that is solely
responsible for the CDE. Again, this
limits the scope of the PCI DSS regulations to just this team.”
It is also important to note that segregation of duties “doesn’t prevent the
cross-functional CDE team from working together in a single space. When
members of the CDE team want to
push a change, they create a ticket to
be approved by the tech lead; otherwise, the code commit and deployment
process is fully automated as with the
main Etsy environment. There are no
bottlenecks and delays, as the segregation of duties is kept local: a change
is approved by a different person than
the one doing it.”
A well-designed platform-as-a-ser-vice (PaaS) can also provide significant
benefits in a highly regulated environment. For example, in the U.S. federal
government, the laws and policies related to launching and operating information systems run to more than 4,000
pages. It typically takes months for an
agency to prepare the documentation
and perform the testing required to issue the ATO (Authorization to Operate)
necessary for a new system to go live.
Much of this work is implementing,
documenting, and testing the controls
required by the federal government’s
risk-management framework (created
and maintained by the National Institute of Standards and Technology). For
a moderate-impact system, at least 325
controls must be implemented.
A team within the General Services
Administration’s 18F office, whose
mission is to improve how the govern-
ment serves the public through tech-
nology, had the idea of building a PaaS
to enable many of these controls to be
implemented at the platform and infra-
structure layer. Cloud.gov is a PaaS built
using mainly open-source components,
including Cloud Foundry, on top of
Amazon Web Services (AWS). Cloud.gov
takes care of application deployment,
service life cycle, traffic routing, log-
ging, monitoring, and alerting, and it
provides services such as databases and
SSL (Secure Sockets Layer) endpoint
termination. By deploying applications
to cloud.gov, agencies can take care of
269 of the 325 controls required by a
moderate-impact system, significantly
reducing the compliance burden and
the time it takes to receive an ATO.
The cloud.gov team practices con-
tinuous delivery, with all the relevant
source code and configuration stored
in git and changes deployed in a fully
automated fashion through the con-
course continuous integration tool.
Going beyond websites. Another objection to continuous delivery is that
it can be applied only to websites. The
principles and practices of continuous
delivery, however, can be successfully
applied to any domain in which a software system is expected to change substantially through its life cycle. Organizations have employed these principles
building mobile apps and firmware.
Case Study: Continuous
Delivery with Firmware at HP
HP’s LaserJet Firmware division builds
the firmware that runs all its scanners,
printers, and multifunction devices.
The team consists of 400 people distributed across the U.S., Brazil, and
India. In 2008, the division had a problem: it was moving too slowly. It had
been on the critical path for all new
product releases for years and was unable to deliver new features: “
Marketing would come to us with a million
ideas that would dazzle the customer,
and we’d just tell them, ‘Out of your
list, pick the two things you’d like to
get in the next 6–12 months.’” The division had tried spending, hiring, and
outsourcing its way out of the problem
but nothing had worked. It needed a
The target set by the HP LaserJet
leadership was to improve developer
productivity by a factor of 10 so as to get
firmware off the critical path for product development and reduce costs.
There were three high-level goals:
˲ create a single platform to support
˲ increase quality and reduce the
amount of stabilization required prior