Tag: metrics

  • Is the software factory working?

    When we replatformed Twill, we built a software factory: agents write most of the code, a merge queue lands it, CI decides what gets through. Which means there’s a lot of activity, and every release I cut (several a week) includes a list of changes. But I need to know if the factory is working. Are we generating impact or just endless lines of code?

    I have a collection of scripts to pull and analyse our output, which I initially put together for a talk. They’ve been useful enough that I keep looking at them, so I added them to the AI dev starter kit. They cover the output and guardrail numbers below; hotfixes and CI spend come from elsewhere.

    The things I think are worth paying attention to: output, reliability, and efficiency.

    Output

    What did we make, and what kind of work was it?

    This chart shows features went from 66% of PRs to 16% while volume went up 24x, and guardrails from 12% to 36%. The scripts classify by PR title, which gave 22% guardrails for September; for this chart I ran a more detailed classification on top. Guardrails should ship as part of the work as well: in September 89% of feature PRs carried their own tests, up from 62% in April.

    With just two engineers, the feature work is where we’re bottlenecked, whereas the guardrail work is more mechanical and can scale much further with orchestration.

    None of this tells you what we actually shipped. That’s what the headline features are for, the same ones that go in the monthly board report. September’s feature share looks low, but actually it was fewer, bigger features – something we changed our process (adding more detailed specs) to support.

    September’s included a member referral programme, marketing pages rebuilt in-app, a Chrome extension for LinkedIn referrals, and a state-of-the-business dashboard. (The extension is in another repo, so it’s not in these numbers.) For things that are internal only, like the dashboard, we shipped early and iterated aggressively with actual users, which means more fixes but also earlier feedback and usage.

    Reliability

    Hotfixes

    Hotfixes are my primary reliability metric: how many issues were severe enough that we shipped an additional release the same day. August had about 10, around half of them in launch week. This has dropped since then, in part due to the increased number of guardrails. September had one. October also had one (already!), but that was a big change (the marketing page rebuild – a bunch of DNS and redirect updates), and I’d planned for that release to need extra attention.

    Guardrails

    The chart above shows the evolution of guardrails. Rules, then checks, then guards, then guards on the agent itself. They reinforce each other: “changes must have tests” was in CLAUDE.md from day one. It wasn’t reliably true until coverage was a required check. That required check doesn’t mean the tests are good, so the agent dispatch instructions now require mutation testing.

    These are the kind of things that used to come out of post-incident reviews, but now they need to evolve continuously. If you’re iterating on your product with AI, you should be iterating on your guardrails too.

    Efficiency

    Median time a PR was open: 1.2 hours in August, 3.5 in September, partly merge-queue contention, partly DB changes (which get more attention and human review) increasing in volume and complexity. PRs landing in two commits or fewer: 67% down to 53%.

    CI spend belongs here too. I have another skill (not public, too tied to our stack) that I use to check the spend is adding value. It should be increasing linearly with output, and where it’s increasing above linearly, the value should be clear.

    More guardrails and more checks mean work is harder to land. That’s a trade, but with two developers incidents are extremely expensive, so keeping them low is the priority. It also means this system can’t scale indefinitely; with one or two more developers, it would need rethinking.

  • A BS Metric for “Diversity”

    The lazy man's guide to Easter.
    Credit: flickr / Daily Marauder

    Recently, I got to use the fact that I am a huge art nerd, to help a team I was on. It was awesome, because I actually felt appreciated – and there’s been something of a dearth of that lately, and because it was a showcase of the positive effects of diversity (something we had been careful to create). When you have people with vastly different lived experiences and interests, there will be unexpected benefits.

    That is the argument for diversity, and why the classic example is that you’d be stupid to build any kind of social experience (especially one featuring location) without the input of women – because they bring a different perspective, one much more rooted in physical safety.

    And you can actually represent diversity as a mathematic model and prove the benefits – as explained in The Difference (Amazon). But one of the points he makes in this book is that it’s not actually about gender, or race, it’s about the way people think. And that there are places where you benefit from homogeneity, because of reduced conflict – and that is in work that doesn’t require creativity.

    The thing is, race and gender can be a proxy for different lived experiences – the personal safety example above – a horrifying number of women have experienced some kind of violence, so chances are, either we have, or someone we know has… and so we worry.

    I’ve had a good experience de-branding myself, and found some benefit to being anonymous at GHC. One of my friends, by contrast, has found that by wearing company and university shirts she is no longer racially profiled as a shoplifting threat – something that had shop assistants trailing her around stores. This is a vastly different lived experience to my privileged little white-girl experience, right? When people follow me around stores it’s because I’m shopping like a fiend and they are helping me.

    So when people see diversity, and see the benefits, it looks like my example above – someone (me) brought a different perspective, there was a clear benefit, everyone was happy.

    Of course, that’s not often what it looks like. In so much of what we do in software, success is invisible, and people only complain when it is broken (not a bad thing – people should be able to expect a good user experience). So a more common case is probably, someone makes a case for something, or does something, and no-one complains about it, hardly anyone compliments it, and it’s easy to ask – did it really matter?

    And when diversity isn’t appreciated, it looks like this – a person surrounded by group think, convinced that the status quo doesn’t make sense, alone, and unheard, wondering if they are going mad.

    From this perspective, it’s clear that micro-management is a terrible thing for diversity (a failure of management in general, but especially in this respect) – because it doesn’t tolerate the slight chaos, and the necessary dissent, or any opinions other than those of the micromanager. A micro-managed culture is by definition a mono-culture, and the measure of diversity in that context, is how many of the people inside it wonder if they are going mad.

    One of the things Unlocking the Clubhouse made me realise, is that whilst university felt like a monoculture, it wasn’t, it was just the group-think was so loud that it was hard to hear the dissenting voices. When I think of my male friends at uni, I suspect most of them came from the around 1/3 of male CS students who don’t identify with the stereotypical CS-culture (as opposed to around 2/3 of women). I’ve been part of a project to try and get that book into the hands of more university students in Australia, and it’s because I think many of them (around 2/3, haha) need to know that identification with the mono-culture is not a prerequisite, it’s a side-effect.

    All of this is a good argument against my current pet peeve – you can’t plonk a couple of new grad women or other minorities in a team and call it diverse. The diversity comes from the dissent, which evolves over time – assuming it is allowed and encouraged, and managed constructively. New grads are less likely to have dissenting opinions, due to having less experience, and less likely to voice those dissenting opinions, due to their lower confidence (especially women).

    The best manager I ever had, assembled a team and made a comment that at the time I didn’t realise was incredibly brilliant. He said that he had expected there would be conflict, but that if we figured out how to work together we would be so much better as a result. And then he helped us figure out our assumptions and all the ways in which we weren’t communicating effectively. And it was so painful to go through that, but he was right – so worth it.

    The worst manager I ever had was a micro-manager with a habit of gas-lighting.

    One of those teams reaped the benefits of diversity, and the other did not. Something that had very little to do with what people looked like (neither team was entirely made up of caucasian males), and everything to do with building a culture where differing opinions was expected and managed.

    The metric of people looking the same is a bullshit metric you can game, at least in the short-term, given enough resources (1/3 of women in University CS do identify with the monoculture, for example). The real metric is people not thinking the same, which as it happens, helps create an environment where the first metric should also do well over the longer term.

    Adding a couple of new grad women on a team does not make for a “diverse” team. Withhold judgement until they have been there for 12-18 months. And then ask, do they have respect, autonomy, meaningful work?

    Because that team may well have developed into a diverse team. It may always have been diverse in opinion, and is now finally reaping the benefits.

    But that is not necessarily true.