ate need for some of that data, you might find a use for it later as new studies come up. Which means you won’t be faced with needing to go back and update your data-collection system to provide for that. The downside is that you will also have all this raw information on your hands that hasn’t been processed for use, which means some engineer is going to have to come along later to build a metrics layer on top of all that. That will leave you with two levels of data—the analytics layer and another layer containing the raw object model data—which people can dive into later if they are looking to get their hands really dirty.

That sort of layering turned out to be a really smart move for us since we now can cater not only to the casual user who simply wants to look at metrics and reviews but also to someone who wants to dive into things.

LP: Are you saying that after you have created these tools for your research purposes, other teams will go on to use them to reflect on their own processes?

CB: Yes. In fact, we did a study a few years ago where we contacted some of the teams that were using our data to discover exactly what they were doing with it, as well as to see whether they had managed to improve the process in any way. We thought that this might be a way to find out where we needed to take our own research.

We found that some teams were
using the data to generate score-
cards, whereas some were using it to
we have found that people will often
open multiple instances of the tool
and then, as they get a bit of free
time, do a small review here and
then another small review there.

So, just because you can see the tool has been open for a certain amount of time doesn’t mean you can assume there has been activity for that whole time. We have the telemetry to determine just how long you were navigating around within the app. That has allowed us to determine that people, on average, spend about 20 minutes per day actively working in CodeFlow—which amounts to a significant amount of time once you multiply that by 40,000 people.

CB: From all that, we have been able to make a number of general observations we’re always happy to pass along as recommendations. In fact, one suggestion I would offer to anyone looking to do something similar to what we’ve done in analyzing their own organization’s code-review process is that, in considering what data to collect, stay as close as possible to the actual object model employed by the application itself. For example, there is almost a 1: 1 correspondence between the tables in our database and the classes in the application. As a result, we didn’t have to think very hard about whether to collect something or not. We just grabbed everything.

So, we ended up collecting all this raw data, and one advantage of that is, even if you don’t see an immedi-

TERRY COATTA
With an eye to the
people outside of
Microsoft that don’t
have your tooling,
do you have any
recommendations
from your
experience that
might prove
relevant?