, ,

Seeing the bottlenecks in the software factory

The baseline of the software factory is output. Ours runs at about 2 PRs an hour, which adds up to daily releases of around 50 PRs (Mondays are often bigger, given it also works weekends).

This is nice, but over the past couple of months I’ve been taking that output as a given, and trying to improve the impact. It’s working: progress on big features has gone up without slowing down the small fixes, and November is on track for roughly double the output of September and October.

From buckets to a river

The challenge is a little bit of non-determinism. It’s not straightforward to know exactly which 2 PRs it will be. That’s a function of priority and what else is in flight, plus some randomness that occasionally results in things just… stalling at 80% for too long.

If software development used to be like carrying buckets of water, now it’s more like a river. The job is still finding the bottlenecks, and making sure the flow is going in the right direction overall.

Recently, that’s meant three changes, each for a different place the flow gets stuck.

1. Stuck at 80%

The first bottleneck I took on was getting it to finish things. That meant building a new habit: finding the things that had stalled at nearly done, and pushing them through to the end.

I went through the GitHub board, and became more stringent about board hygiene. I also completely cleared out the ready column. I was travelling and not generating new work, so it became much clearer what was stuck.

2. Too big to move

Then, for bigger work, we adopted a spec-driven development approach that I think is pretty standard. Large features now live in a “needs spec” section on the board, and get:

  • A spec for human review, ideally with wireframes
  • A designated Claude as product owner (a pinned Claude chat, separate from the orchestrator)
  • A feature flag
  • Slicing into pieces for orchestration
  • A rollout checklist

This made large chunks of work much easier to move through. Rather than trying to keep it all in my head, or worse, unravel Claude talking to itself across a dozen GitHub issues, I have an abstraction over all of that.

The product-owner Claude owns the overall feature and has the context about it. That means I can query where things are, and it has clear asks of me. It’s also where a more expensive model, Fable, earns its keep.

3. Out of sight

What was still missing was an easy view of what is where, and what is blocked on whom. The higher the volume, the harder this gets. I post release notes every day, but at 50 PRs a day, that isn’t always easy to follow.

So, two new views:

  • Mine is what I care about: moving my big initiatives along, and keeping the orchestrator unblocked.
  • The team’s is their point of view on what matters: the monthly goals, and all the little requests.

My view

Priority work on one side, questions on the other. Each question comes with options to tap, and the orchestrator acts on my pick the next time it checks in. Decided questions move to a log, with a line on what was done about each.

Before, its questions came floating in the chat without context and often got lost. This way, it’s easier to batch, answer the quick ones quickly, and have time to think about the others.

The orchestrator view: priority work tracks with their status and pieces of work on the left; questions waiting on me, with options to pick, and a log of decisions on the right.

The team view

The goals we committed to in the monthly board update, the smaller work people asked for, and questions grouped by who needs to answer them. It refreshes at 14:00, 18:00 and 22:00 my time, through the US working day, and answers go back onto the GitHub issue at the next refresh. Anything built goes to “Check on staging” first, so the person who asked for it can try it before it’s turned on in prod.

It also shows how many of their requests we get through. People are inclined to fixate on the thing they want that we haven’t built (yet); this puts it next to everything we have.

The team view, in three columns: monthly goals with progress bars, work people asked for grouped by status, and questions for each team with options to answer.

Both screenshots are anonymised and compressed versions of the real thing.

A day in the software factory

The best way to illustrate all this is to outline a day in a software factory, as a European working mainly with Americans. I don’t work on this full time, but when I do a full day, this is what it looks like. It’s also only a partial view: the other engineer has her own, works more on bigger features, and validates DB changes before they reach staging.

The orchestrator runs all the time. Between every block, I check in on it for 10 minutes, tops, to make sure its questions are answered and it has work to dispatch. (Honestly, I also check in on it out of hours – like right before I go to bed 🙈.)

A timeline of a day from 7am to 6pm Cork time: the orchestrator running throughout, Claude preparing the release from 7am, release from 10am taking 30 minutes to 2 hours, short orchestrator check-ins between blocks, large features and triage late morning, lunch, a proactive block from 2pm, and time with the US team from 4pm.

Every morning starts with a release, which Claude starts preparing at 7am: it checks and dispatches any must-fixes, and runs all the checks. The release itself takes 30 minutes to 2 hours, depending on how big it is. That covers reviewing Chromatic diffs, a final check on DB changes before prod, checking everything goes live as we expect, final validation and switching on anything due to go live in prod, and editing and posting the release notes to the team.

Then it’s the large features in flight: what is unblocked, what the next steps are, what needs to be validated and what decisions can be made. After that, the team’s feedback from their day (my overnight) gets filed into the ready column for the orchestrator, or needs spec for everything else.

After lunch is a proactive work block, for things like:

  • Working through the needs-human column
  • Putting together specs for other features
  • Iterating on and refining large in-flight features
  • Going through the large in-flights, and figuring out what will move them forward
  • CTO work (strategy or being the IT department, mainly)

Then from 4pm, the team is online: meetings, ideas coming in, filing issues and technical support.

What’s next

My current unsolved problem is how many epics we can really have in flight. Or perhaps the real problem: I got so excited by this system that I dispatched too many epics. More generally, now that we have a new baseline, I’m working out how to iterate as we build larger features.

I don’t know exactly what the next blocker will be. The system is still settling, and I’m sure it will emerge. Part of it will likely be CI speed – the increased output already made speeding up CI necessary, so we moved to 8 cores and cut our CI time.

go deeper

Navigating the AI Shift

From anxious and reactive to genuinely capable. Build real fluency with AI — without the hype.

Comments

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.