otherwise be using to create code. If developers are rewarded only for adding functionality, that’s going to end up crippling the code-review process, which in turn will almost certainly have an adverse effect on the maintainability of the code that’s generated.

CB: One thing I would like to add is that the code-review process we now have at Microsoft has more or less grown organically—through experimentation—from the grassroots. I mention this only because I think it might also work well for smaller companies, instead of having some process that’s mandated from the top down.

Also, each product group at Microsoft does code reviews a little differently, with each group using its own set of policies that have essentially come together organically. While this probably won’t come as a ground-breaking revelation, it can definitely be said that there is no one-size-fits-all solution for code reviews. This only serves to reinforce the importance of being willing to let your approach evolve organically such that it ends up fitting in with your work processes with the least amount of friction while putting the lightest burden possible on your developers.

Another important point is something Michaela talked about earlier, which is that treating code review as a first-class citizen—just as many companies are likely to treat testing—is probably the best way to get the most bang for your buck. If, instead, it becomes something you are just expected to do, like flossing your teeth daily, then you’ll find people aren’t going to embrace it. But if you say this is important and so will be tracked and evaluated, then people are likely to respond to that. Certainly, that’s how it has worked out here.

And then the other thing I would add is that it’s instructive to think in some depth about what it is you’re really looking to get out of code reviews. Then, of course, you should also think about how you can go about measuring that. To the degree that you can track those metrics and set targets, you’re always going to achieve more.

 

© 2019 ACM 0001-0782/19/2 $15.00.

to workflow. Still, there are times when two unrelated concerns end up getting crammed into a single review, so we are hoping we will be able to untangle some reviews by automatically splitting those concerns into two smaller reviews. On average, that ought to lead to better turnaround times, as well as better outcomes.

LP: Have you taken any steps to get development teams to focus their code-review time on correctness and content versus style? Have any tool changes or process changes been implemented toward that end?

JC: We haven’t done a proper study of that, but there is a team here that’s done something along those lines. This is something that had to do with some factoring changes they considered to be low-risk—such as the renaming of methods or local variables. For example, this might involve putting a special tag on a review to say, “We don’t really need to have two people look at this. One is enough since it’s very unlikely we’ll have any functionality issues here.” Modest as that might seem, it can also prove profound since it turns out there are many changes like this floating through a legacy system—clogging the system.

The thing to remember is that it’s not just about making one change go faster, since what you’re dealing with here is a pipeline of changes— meaning that any change you can redirect to a lighter-weight path is going to lower the load on your key people and get it out of the way of other changes waiting to be reviewed. That’s just the sort of thing that makes for a more efficient system all the way around.

TC: With an eye to the people outside of Microsoft that don’t have your tooling, do you have any recommendations from your experience that might prove relevant?

JC: I would say the one thing to
recognize is that comments related
to maintainability are primarily what
you are going to get out of the code-
review process. Contrary to popular
opinion, locating bugs is not the pri-
mary outcome. The other important
thing to bear in mind is that the small-
er a review is, the better it’s going to
be. In our case, we’ve found that if a
review contains more than 20 files,
it’s too big already. In fact, from our
study of all the data at our disposal,
we’ve concluded that for more than
20 files the density and usefulness
of comments degrades significantly.
This is actually more a rule of thumb
than a precise limit, but it is useful to
keep in mind.
Also, if your organization has data
from past reviews, I would suggest in-
vesting in a recommender system that
can help make some of the administra-
tive steps a little less tedious. You can
even use these systems to automatical-
ly address some of your maintainabil-
ity issues, which is something we’re
starting to get into these days. That is,
you can imagine that some of these
maintainability issues are essentially
things that might be autodiscovered
and flagged, which means you then
don’t have to expend any human re-
sources to get this accomplished.
Another thing, as we just dis-
cussed, is the idea that two signoffs
on every change might be too many.
If you look at the distribution of com-
ments made by either the first or the
second reviewer, you’ll find that your
first reviewer typically discovers the
most egregious problems. In many
cases, waiting for a second reviewer
to corroborate those findings before
allowing the commit into the main
source tree might be less efficient.

MG: My biggest takeaway from the survey is to always make the burden of code reviews just as small as you possibly can. Part of that comes down to having a good code-review process that enables and encourages comments that can be easily reviewed.

Another important consideration has to do with supporting the reviewers themselves by giving them advance notice about any reviews that might be coming up and giving them enough context so they will be able to dive right into a review without having to figure all that out for themselves. Doing what you can to reduce the size of reviews can also be helpful. But I think what is really important is to make the reviews just as uncomplicated as possible, since, otherwise, you may end up with reviewers who have no clue about where even to start.

Also, organizations need to show they recognize the value of code reviews since there’s no question that they take away from the time developers could