ate need for some of that data, you
might find a use for it later as new
studies come up. Which means you
won’t be faced with needing to go
back and update your data-collection system to provide for that. The
downside is that you will also have
all this raw information on your
hands that hasn’t been processed
for use, which means some engineer
is going to have to come along later
to build a metrics layer on top of all
that. That will leave you with two levels of data—the analytics layer and
another layer containing the raw object model data—which people can
dive into later if they are looking to
get their hands really dirty.
That sort of layering turned out to
be a really smart move for us since we
now can cater not only to the casual
user who simply wants to look at metrics and reviews but also to someone
who wants to dive into things.
LP: Are you saying that after you have
created these tools for your research
purposes, other teams will go on to use
them to reflect on their own processes?
CB: Yes. In fact, we did a study a few
years ago where we contacted some
of the teams that were using our data
to discover exactly what they were doing with it, as well as to see whether
they had managed to improve the
process in any way. We thought that
this might be a way to find out where
we needed to take our own research.
We found that some teams were
using the data to generate score-
cards, whereas some were using it to
we have found that people will often
open multiple instances of the tool
and then, as they get a bit of free
time, do a small review here and
then another small review there.
So, just because you can see the tool
has been open for a certain amount
of time doesn’t mean you can assume
there has been activity for that whole
time. We have the telemetry to determine just how long you were navigating around within the app. That has
allowed us to determine that people,
on average, spend about 20 minutes
per day actively working in CodeFlow—which amounts to a significant
amount of time once you multiply
that by 40,000 people.
CB: From all that, we have been
able to make a number of general
observations we’re always happy to
pass along as recommendations.
In fact, one suggestion I would offer to anyone looking to do something similar to what we’ve done in
analyzing their own organization’s
code-review process is that, in considering what data to collect, stay as
close as possible to the actual object
model employed by the application
itself. For example, there is almost
a 1: 1 correspondence between the
tables in our database and the classes in the application. As a result, we
didn’t have to think very hard about
whether to collect something or not.
We just grabbed everything.
So, we ended up collecting all this
raw data, and one advantage of that
is, even if you don’t see an immedi-
With an eye to the
people outside of
Microsoft that don’t
have your tooling,
do you have any