new products and features are delivered?
˲ Comprehensiveness. Is enough data
captured across all systems of record?
For example, to measure time to market for a customer request, you may
need data from a customer/support
tracking system, the roadmapping/re-quirements system, the agile tool, and
the deployment tool chain.
˲ Correctness. Is the data sufficiently
correlated to be correct? For example,
if a support ticket and a defect are actually the same item but exist in two
different systems, should the two systems be integrated in a way to indicate
that these are the same item, or do you
risk double-counting defects in this
System Data Advantages
˲ Precision. Only system-generated
data can accurately show minute, second, and millisecond response times.
˲ Continuous visibility.
System-generated data is particularly well suited
for continuous/streaming data and real-time reporting. You can just point it
to the data store and gather everything
for targeted analysis later.
˲ Granularity. Data from systems
can provide very granular data, allowing you to report on subsystems and
components. This is useful for identifying trends and bottlenecks, but
requires additional effort to create a
higher-level picture of the full system.
The more granular the data, the more
work is required to paint a full picture.
˲ Scalability. Once the integration
and visibility infrastructure is implemented, it can be pointed at all systems. This means that the solution
can be scaled from getting visibility on
a single project to dozens or hundreds
of projects with large amounts of data.
To use an analogy to illustrate:
when building a house, a contractor
may use concrete for the foundation;
wood/nails/screws/drywall for the
walls; wiring and plumbing; brick for
the exterior; paint/carpet for the finish; plus any materials for the kitchen
and bath. In order to track and monitor progress, you build in monitoring
to track each piece of the construction
and install it as the house is built.
Once installed, each and every piece of
this infrastructure (specific data) can
continually provide reporting and
metrics (continuous data) at subsec-
ond intervals (precision). You can then
combine and correlate (volume and
scale) these to create a full picture of
what is happening in your house.
System Data Challenges
˲ Capturing behavior outside of the
system. This may be the most important yet most overlooked limitation in
system-based data. An example is version control: your system can tell you
only what is inside of it. What portion
of the work being done is not being
checked into a version control system?
Common culprits include system configuration and database configuration
˲ Gaining a holistic view. Eventually,
system-level data can provide a relatively full view of your system, but this
requires full instrumentation, plus
correlation across measures and maturity in reporting and visualization
techniques so that teams can understand system state. This is a nontrivial
task, especially if undertaken without
the right tooling and infrastructure in
place. Additionally, the holistic view
should include the human aspects of
the process, such as the difficulty of
deployments and software sprints,
which are important for understanding the sustainability of the work.
˲ Capturing drifts in the system. If any
part of your system stack changes and
your data collectors are not updated,
your view of the system will be inaccurate. Note that this is not a characteristic of a first-class data reporting
solution, but it happens in some commercial systems and in many home-grown solutions, so it is worth mentioning as a condition to watch for.
˲ Cultural or perceptual measures.
If you want to measure aspects of culture, these are perceptual and should
be measured with surveys. Further, any
measures that come from system databases (such as HR systems) are usually
poor representations of the data you’re
trying to collect and will be lagging indicators. That is, they will be able to
measure something only after it has
happened (such as someone leaving a
team or an organization). In contrast,
survey measures can let you measure
perceptions of culture in time to act on
System-based metrics are useful,
but they cannot paint a complete picture of what is happening in your software-delivery work. Therefore, it is
strongly recommended that you augment your metrics with complementary survey measures.
Survey-based metrics generally refer to
data about systems and people (such
as culture) that comes from surveys.
Ideally, these surveys are sent to the
people who are working on the systems
themselves and who are intimately familiar with the software-development
and delivery system—that is, the do-ers. It is better for teams to avoid surveying management and executives,
because, as a recent study by Forrester
shows, executives tend to overestimate
the maturity of their organizations.
Important aspects of this data include:
˲ Cohesiveness. Survey-based data is
particularly good at providing a complete and holistic view of systems. This
is because it can capture information
about systems, processes, and culture.
Measure your system periodically and
at regular intervals: every four to six
˲ Correctness. Survey design and
measurement is a well-understood
discipline and can be leveraged to
provide good data and insights about
systems and culture. By using carefully
designed surveys with statistically valid and reliable survey questions that
have been rigorously developed and
tested, organizations can have confidence in their survey data.
Survey Data Advantages
˲ Accuracy. When collected correctly, survey data can provide accurate
insights into systems, processes, and
culture. For example, you can measure system capabilities by asking
teams how often key tasks are done
in automated or manual ways. When
designed correctly, this provides a
fast and accurate measurement that
can be used to baseline and guide improvement efforts.
˲ A holistic view of the system.
Surveys are particularly good at capturing
holistic pictures of systems, because
the answers that respondents provide
synthesize data related to automation,
processes, and culture.