Accidentally in Code

I keep hearing complaints about code review. About volume, about quality, about trying to take responsibility for code produced by colleagues who have abdicated their responsibility for thinking and are uncritically hitting enter.

But how much of that is just grumpiness? Frustration? “AI slop” is the new “I didn’t like it.” Code review was always something I paid attention to as a manager, because when there was conflict on a team, it was the place most likely to show up. Code review is subjective work presented with the illusion of objectivity; everything unsaid can play out in it.

In the software factory model, no one writes the code and no one reviews the code. These go together: once the code is generated, human review is the bottleneck, so you validate some other way, through guardrails and automation. Most teams aren’t building from scratch, though. Akin to the challenge of self-driving cars, they are on mixed roads, humans and machines sharing the same lanes, which is the harder case (I loved this podcast on the topic).

So I wanted to see whether I could reverse engineer code review: some kind of machine gate a PR has to pass before it’s allowed to request actual human time and attention. Much of my career has been in open source, the write-code-in-public kind, so I picked a team with an open source repo I knew well enough to have context. I deliberately didn’t pick the highest-performing team I’d worked with: this one had mixed levels, engagement, and AI adoption, and I was curious how that played out in review.

What I did

First, I constrained Claude to be read-only on GitHub. The last thing I wanted was to leave a trail of inadvertent and likely weird AI-generated comments.

Then I had Claude (Opus) find the strongest code reviewers on the team, seeded with three names I asked it to confirm or deny. Its reasoning tracked with what I knew. It also flagged some reviewers who were thorough but inclined to nitpick, which made me add a style guide to the scope.

An aside on style guides: I started my software engineering career at Google, where the style guide was canon. I had readability in two languages, Objective-C and Java (basically, I jumped through a set of hoops to prove I could deliver high-quality reviews according to the style guide). I don’t really subscribe to any religions, in software development or otherwise, but one thing I took from that period of my career is that the true benefit of a style guide is that we do not discuss it, and spend the time on the things that matter instead.

Claude (Sonnet) built a corpus of 4,176 review comments and summaries by twelve reviewers, on over 1,700 PRs going back about 20 months. A Fable agent per reviewer pulled out the patterns in what they flagged, with examples: ~300 of them, which another agent compiled into 67 themes. Then I had it turn the themes into checks. Finally it went through every PR merged since the start of September (~500) to work out which checks the version the first reviewer saw would have failed, and whether review caught it. As on many teams, there were already AI review bots on the repo, so “review” in everything that follows means humans and bots together, unless noted otherwise.

This took real time, most of it reading what came back. It’s the kind of data I would have loved to have when I was running large orgs, and now it’s feasible.

What the machine recommended

What came out was a tidy recommendation: 57 checks:

• 11 style guide rules
• 8 test requirements, either tests for classes of issue or required test types for certain kinds of work
• 26 CI and static analysis checks (lint rules and scripts that run on every push)
• 12 instructions for a project-specific review skill that an AI reviewer could run, the judgement calls that don’t reduce to a regex.

I’ll call those style, tests, CI and judgement from here on. It also set aside 16 norms that authors had pushed back on at least once, sometimes for good reasons, and said not to enforce those mechanically at all.

Run against the last six weeks, more than half of PRs would have failed at least one check, with ~700 issues between them. 71% of those were never raised in review, by human or bot, and most of those merged without anyone mentioning them. Review raised 75% of the judgement issues, for example around metrics and concurrency, 32% of the style issues, 14% of the missing tests and 10% of the CI-level problems. Count only humans and it’s 43%, 22%, 9% and 5%.

(Some caveats. The CI checks were simulated by agents rather than actually run. The agents could see the review history, which probably flatters the judgement numbers: the checks were mined from these reviewers’ comments, so “review raised it” is partly “the check describes what the reviewer said”. A second pass re-checked every eighth PR and agreed with about nine in ten of the findings it was confident about. And only GitHub is visible, which is obviously incomplete.)

On paper, that’s the case for a gate. Write the bar down, have an AI teammate check every PR against it, and only then ask for human time.

Very nice. But what does it say about code review?

What it taught me about code review

It’s inconsistent. 92% of PRs had exactly one human reviewer (6% had none), so whatever that person missed went through. Individual reviewers raised anywhere from none to 38% of the problems in the PRs they reviewed, with a median around 20%. And there were 65 cases where someone flagged a problem on one PR, then let the same problem through on another. Reading the examples, my impression was that the miss was often a side change rather than the point of the PR. That makes sense, and with more and bigger PRs, it’s a prime reason to move this work to automated checks.

Some of that is just asking people to do a machine’s job. PR review was never the way to catch mechanical things. How do you hold test coverage in your head? That’s why teams that cared about these things always instrumented them. Looking back, readability at Google was kind of weird: a linter installed in my head. Test coverage requirements would probably have been more impactful.

The misses don’t obviously cost anything. I had Claude trace the 400+ unflagged issues forward: did a bug fix, hotfix or revert later touch the same lines? Then it compared that against other new code in the same PRs, adjusted for size. About the same. 20 of the 345 it could trace were followed by a fix, against about 17 expected. Reading the 20, only about half fixed anything the check plausibly described. The clearest cases were missing tests, which fits my hunch about coverage, but on small numbers to YMMV.

That’s a lower bound: each PR had between two and 42 days of follow-up. It also only measures bugs, not the slower cost of architectural mismatches that make the next change harder to test, or dead code someone reads around for years. But it was enough to shake my intuition that the misses mattered.

So what is review doing? Enforcing things the team hasn’t necessarily agreed to. Some of what reviewers raised was really shared: everybody flagged leftover dead and debug code. A lot of it wasn’t. Dependency injection, the architecture rule that came up most, was raised by only half the reviewers. Asking for missing tests was something seven of the twelve did over the 20 months, but in the six weeks I audited, review asked for tests only about 14% of the time, mostly humans. And in 16 places a reviewer’s norm got pushed back on, and in a few of them reviewers wanted opposite things: for instance, whether a feature flag should be read at runtime or once at launch.

These are all reasonable stances to take, and it’s possible they were situationally dependent in ways that I don’t have insight into. But from this data, and for an individual onboarding, it would be hard to distinguish a team norm from one person’s preference. It’s muddier still when, as here, some of what read as a reviewer’s judgement was actually AI output transferred via C+P.

That’s where the conflict comes from, and where bias gets in. Google studied pushback in its own code review and found the amount authors get varies with their gender, race and age, at a cost of more than 1,000 engineer hours a day, borne by engineers who aren’t white men. It is very easy to shred a PR if the person who wrote it seems like someone whose competence is in question, whether that’s down to what we politely call “unconscious bias”, because they were annoying in the meeting yesterday, or because the perception is they have handed their judgement to the machine.

So would I actually implement this?

There’s a real cost. An AI review on every PR costs tokens, and new lint rules and required tests add CI minutes to every push. The tokens are largely a more effective use of the AI review that’s already in place, at about a dollar a PR. Implementing everything would add somewhere between 6% and 38% to the CI minutes the repo already uses, depending on how carefully the expensive checks are scoped. Then there’s time. A gate adds 5 to 30 minutes before a PR goes green, a few percent of the time a PR already waits for its first human review. The bigger cost is that 55% of PRs would fail a check on their first push, so more than half would go round a fix-and-rerun cycle before a human was even asked to look. Against a bug rate that, so far, doesn’t move, that’s hard to justify on bug prevention alone. The case for it is consistency, which I think has value of its own, but has to be balanced against the cost.

When individual enforcement becomes team enforcement, the team needs to agree. The cheap checks, which are also the lowest leverage, are the ones nobody disputes. The highest-leverage ones, like dependency injection, or what counts as properly testing a feature flag, are the ones that need an expensive team discussion first. Skipping that discussion would turn one reviewer’s preference into a CI failure. But the value of that discussion could easily lower than what it would cost to have it.

The biggest value of code review isn’t in the repo. The most valuable thing review does is ask whether we’re building the right thing in the right way. Nearly half of what review raised here didn’t map to any check: product and behaviour questions specific to that PR. Ideally, those come up before review, because review is the last gate before user impact. But they’re context-specific, and the data to answer them isn’t in the codebase. What they point to is structure: who gets to decide what. What can an IC decide about their own project? What needs a broader group? Google’s readability and owner split was a version of this: style was settled once, for everyone, and ownership said whose call the rest was. Without that structure, those decisions get made in code review, by whoever happens to be reviewing.

So, would I implement it? The uncontested checks, yes: done efficiently, they add only a percent or two to CI. The rest depends on decisions the team hasn’t made yet, and depending on the team those discussions could be expensive. Right now code review is where those decisions get made, one PR at a time, by whoever is reviewing. Even if that worked, more or less, when humans wrote the PRs, I don’t think it works in an agentic world.