Accidentally in Code

I’m not someone who gets called when things are going well. So when Michelle – the Twill CEO – called me in December, it was because she needed some help with engineering. In 18 months of being an advisor, this was the first time it was a phone call and not an email.

There were a lot of things going on, but at the core of it was the data model, which was starting to really hold things back. The data layer was more like CSVs than a proper relational database. The no-code platform was good for getting to product-market fit, but it was clearly hitting its limits.

Twill is a two-sided marketplace focused on hiring – flipping it from high-volume, low-signal to a low-volume, high-signal model. Companies can post jobs, and members can refer their network to jobs with personal recommendations, and get paid for placing people. I had been a hiring manager in the agentic era, inundated by applications from agents, trying to sift through and figure out who would be good at a time when it seemed that every interviewee was extensively prepped by AI and giving bland answers. The idea of using human networks and knowledge about who actually does a good job was – is – compelling.

My first job was to rationalise things a bit (no more two-way writes between Bubble and Airtable) and then get it to a place where the business could function on it whilst we moved to a more normal setup – one with a proper database. That took our one full-time engineer until April, at which point I was able to increase my hours.

I knew agentic coding could speed things up – I’d just got started with making a slightly unhinged side project that I never would have prioritised before Claude was a good frontend programmer. My previous team had been using AI code reviews since early 2025. I could see the utility, and some of the potential, but I had no idea how to estimate in this new world. I wrote a migration plan, textbook, the way we used to – decompose things carefully, derisk, run things side by side.

This is a story about how we threw that plan in the bin and shipped a full replatform in four months.

This is also a story about how I went from running an 80+ person team to running a software factory. Many things have changed, but it turns out that running a large team was a better setup for this shift than I thought.

In my previous life, I was one level below VP at DuckDuckGo, responsible for roughly half the engineering org – mainly client (iOS, macOS, Android, Windows, C++ with some Frontend). My background is in native application development. Now, it’s me, and one other engineer, building a Node.js app on top of a Go API. I am – somehow – responsible for all our AWS and other infrastructure.

The challenge of the V2

V2s are notoriously difficult. I know, because I was the kind of leader often dispatched to take the failing V2 project and get it delivered. Amongst other things, I’ve led teams responsible for delivering text editors and web browsers – incredibly complex applications where reliability is fundamental.

If “underpromise and overdeliver” is the rule to live by, V2s are where that rule most often falls apart. V2s are systematically overpromised and underdelivered. Because why build a V2 if it’s not going to be better than what you have currently? But also, replacing what has been built up over a period of years is incredibly difficult.

V2s fail because it’s so hard to define the scope, and stick to it. Because it’s easy to align people around a dream that will deliver everything everyone wants (eventually), and much harder to deliver according to a strategy where everyone is going to be explicitly disappointed on some shorter-term timeframe.

The old platform was functional, but expensive to run, especially the kind of data work that was critical for an efficient user experience. The structure of the data was fundamentally incompatible with the kind of AI roadmap every startup is tasked with delivering.

First, we stabilised the old platform and made sure critical functionality for business operation was in place. In April, we started building a new system. We started with a list of API calls for each screen, and the DB model backing them. This was foundational information for a rebuild, and relatively efficient to extract. On top of this information, I focused on infrastructure – building a design system, and a Go API – whilst the other engineer worked screen by screen, rebuilding the platform.

The first decision we put in the bin was running the backend on AWS, and instead running it on Render with the database on Neon. The complexity of AWS was just not worth it for such a small team, and the cost savings – if any – were marginal. This was a big unlock, because instead of futzing around with Terraform, in a couple of days we had everything running end to end. We kept AWS for S3 and can move to it at a later date should it become worthwhile.

The second decision we put in the bin was the side-by-side operation. This came back to the data model, which I’ve been comparing to putting your to-read pile in a library system. Painful, and obviously a one-way door. You can’t run both systems side by side when one of them runs on a database and the other runs on a jumble sale. At this point, it was clear that we would deliver less incrementally, and less safely, than I had planned or would want to.

The third decision we put in the bin was the launch date. We had no idea how to estimate, so had picked July 4th as a suitable date – what better time to take the site down than when the user base is having a long weekend? We were basically feature complete then, but given the one-way door of the launch, we pushed back about a month (to August 2) and spent that time on data migration, cleanup, and hardening.

All this time, I was saying no to every feature request, and every bit of scope creep. Michelle put me on blast on LinkedIn. I held firm. You can only say no for so long though. We really needed to ship.

On August 2, we shipped. That week had some firefighting, but our hardening and process period paid off. The worst thing that week was a number bug – cents as dollars. Which, whilst embarrassing, at least wasn’t anything safety critical given the extent of PII in the platform.

Leaving the no-code drag-and-drop style platform is always going to be hard. What’s interesting to me is that it’s now possible. The replatforming was obviously needed, but how do you possibly do it with two people, one part time? How do you justify the cost to the business? Claude made the math make sense, and the software factory model meant we could ship in four months.

Building the factory

I was doing what I used to do – paying attention to what was working, what was not, how problems slipped through. I set up infrastructure, CI, prioritised work that eliminated problems. I built process, but this time in the form of skills rather than spreadsheets or checklists. Together, my teammate and I built a software factory.

Michelle always used to tell me one day I would be her CTO, and I always demurred. I didn’t think my skillset was what she needed – I thought she needed someone who had more experience in web apps and backends and infrastructure. Someone who had more of a tech lead manager mindset, capable as a people manager, but keen to still write and ship code. Turns out, she was right and I was wrong. I am not the profile I would have recommended for her before the agentic era, but I was well suited to build a software factory. Managing 80+ engineers, I didn’t write the code. I built the team that wrote the code, and the processes that made them effective.

One of the differences for native application teams is that we’re not typically supported by shared service teams. “Infrastructure” and “SRE” teams usually focus on supporting backend and web. We build our own CI systems, do our own releases, and handle our own incidents.

Whilst feature teams tend to be divorced from the underpinnings of DevEx and reliability, we are not; we build them ourselves. We also tend to feel the cost of problems much more acutely – in part because of this, in part because of the feedback loop of the app store ratings. As such, I was used to owning and iterating on quality issues. It had been my job – admittedly in a very different context – for years.

I was on a learning curve, trying things, asking questions, seeing why things didn’t work the way I expected them to, and iterating. Eventually I ended up at orchestration.

Orchestration was a result of DDoSing myself. Too many things in flight, flitting between them, nudging them along. So many terminals I needed to colour code them in order to tell which was which. Burning CI minutes on collisions.

The trap I fell into is that the cost of generating drops, and you can have a lot more things going. Just because you can doesn’t mean you should. Instead of nine direct reports, I had nine terminals. But similar to having that many direct reports, the level of attention needed by each varies, and a system for that helps apportion focus.

Orchestration was part of fixing that, living in three terminals:

• Terminal one – sequential focus on things labelled “needs human”.
• Terminal two – the orchestrator, 3-5 PRs in flight at any given time.
• Terminal three – the merge queue, final validation and integration.

This unlocked momentum, and with that momentum more guardrails became necessary – and also cheaper, because a lot of that kind of work is mechanical and well suited to orchestration. I know there’s debate about the utility of PRs, but I still believe PRs are useful. The PR itself is a queryable unit over which you can run rules and reports.

Having hit momentum and feeling happy with the system, Jean – my collaborator on DRI Your Career – wanted to try it. So I retrofitted a couple of months of learning onto another platform. Jean and “intern Claude” had mostly built it. It was time for Claude to be promoted from intern to engineering team.

I used to scale teams over a period of years. Here, in a week I put in:

1. A release branch and release process. This would have been overkill – a bit anal – for two people, but it’s the natural progression for a team no longer wanting to deploy every PR on main. I know this goes against conventional wisdom, and you can probably see my background in native app development here, but for a two-person team with no on-call SRE, better to schedule your problems for a convenient time.
2. The AI-native piece (orchestrator and merge queue). Which also lets Claude work overnight while we’re sleeping and don’t need the tokens for everything else.
3. Board hygiene – P0/P1/P2/P3 aligned with orchestrator + merge queue. The board setup of a well-organised 6+ person team, which keeps the orchestrator (and us) focused on impact rather than maxing out token and CI spend. Features slated for the next release get tagged – I brought this in when I realised we could easily ship >50 PRs without a finished headline feature.
4. DevEx and optimisations – ephemeral DB targets, authenticated E2E tests, env var requirements, a full design system, automated nightly DB backups. Things we hadn’t needed when validation was manual, but that are helpful to have in place before leaving an AI unattended. A level of DevEx I used to watch 20+ person teams struggle to reach, which I actually consider not over-engineering but necessary to support the velocity.

It used to be the case that there was a 2-5 person team process, then the 6-10 range, then the 20+, the 100+, the 1000+. Managers got hired in part to deploy the playbooks that moved teams through those ranges.

All but one of these are things I’ve done as part of scaling a team in the before times. I just didn’t use to do them all in a week. This was the moment I realised that the playbook is not gone – it’s reconfigured. Smaller teams have everything they need to be effective with agents, because almost all of it is stuff that bigger teams used to put in over time.

I have heard managers say this shift doesn’t impact them the way it impacts ICs. It does. Even if your process was right-sized for your team before, it isn’t – or shouldn’t be – once each person on your team is running a team of agents.

Quality control

Like many software engineers, I’d noticed how much worse software seemed to be working overall. I’d seen the gaps in reasoning that happen between Claude and humans. Claude reasons about code. Humans reason about what is in front of them visually. It was clear to me that quality control was critical.

As engineers, we plan what we expect to build, and we expect some work to come in to maintain it. Incidents, bugs – the second roadmap. The one we don’t write down, except when we’re explaining why the first roadmap wasn’t fully delivered.

We’ve all seen the news about Meta’s plan to replace staff with AI, and how it was scaled back. Their own data shows that whilst code changes to internal platforms and infrastructure went up 220%, there was only a 36% increase in changes reaching users. That’s comparable to the increase in major incidents – 40% – and the time spent firefighting them went up 70%. (Reuters special report, Katie Paul and Jeff Horwitz, 26 Aug 2026)

Increased output is not the problem. The problem is how to turn increased output into business value.

Which brings us to guardrails. I’m hearing this word a lot, but it rarely includes a definition. Here’s mine.

A guardrail is a way to prevent problems.

This is intentionally broad, because what constitutes a sensible guardrail varies. We’re talking about them a lot because that has changed so much in the agentic era.

Large companies always built layers so people could operate with partial understanding without creating catastrophic problems. The practices always existed. But smaller teams relied on things like “tribal knowledge” or “only hiring seniors”.

Automated code reviews were of course an initial guardrail. I quickly learned that putting things in skills files was good, but not sufficient. For robots, like humans, written good – automated better. I became fascinated with what percentage of output should be guardrails.

The good thing about guardrails is that they are often quite mechanical, and once I had a good orchestrator and merge queue set up, I could throw that kind of work into it more easily. I would use my oversight – noticing problems, additional PR reviews, the things that almost shipped but didn’t – and extract from it the preventative work that could be dispatched via orchestration.

This was also a benefit of building from scratch. Legacy codebases contain competing patterns and ideas, and half-finished migrations. In a new codebase, we can decide: this is how we build. With agents, we can fix the mistakes we made as we spot them.

This is where I think the drive for hands-on managers actually has value – because the volume comes faster, and you can’t evolve these checks in post-incident reviews any more. You have to be on them constantly. If no one writes the code and no one reads the code, how do you figure out what is needed before the incident?

The chart below shows our guardrails evolving over the course of the build. Starting with written-down instructions, and increasingly automated over time. Because it turns out, agents also don’t always RTFM.

Despite “changes must have tests” being in CLAUDE.md from day 1, that didn’t reliably happen until test coverage was instrumented and a required check. The .md rules and automations are not divorced from one another; they can reinforce each other.

• Mandating 70% test coverage doesn’t mandate quality of tests – enforcing mutation testing in the orchestrator does that. Every agent must “break the fix, show a named test fails, restore, show green – and put that evidence in the PR body.”
• Every diff gets asked “does any check here report success without verifying what it claims?” Before a PR is bounced, a second independent agent re-verifies the finding from file:line only.
• Any bug that reaches production requires both a regression test and a monitoring alert.
• An agent hook, block-migration-merge.sh, stops Claude merging any PR that touches a migration – these changes are hard to roll back and can impact the integrity of the data, and so a human needs to take responsibility for them.

Before, the discipline was: you found a bug, you wrote a test so it wouldn’t come back. Good practice. Now, I do that, and then I ask: how do we guard against this class of problem? Because competent humans rarely make the same mistake twice. AI will make the same mistake endlessly.

Making a class of issue impossible is not cheap. This is why we didn’t use to do it – we relied on the incident postmortem and code review, and the visceral shared memory of the team about that time we messed up. Now, a minor bug spawns five-plus follow-ups – graduated solutions of increasing complexity.

For example:

• Aug 26: the internal candidate drawer passed user_account.id where candidate.id was expected. Both integers – so had they collided, it could have silently loaded and written another person’s data. Staff-only, caught by an alert, and fixed within hours.
• Aug 26: named the class – “silent wrong-entity resolution”.
• Sep 2: ID-space collision detector; test-DB sequences offset so the two ID spaces can’t coincide in tests.
• Sep 23-26: typed IDs at the sqlc boundary – passing the wrong ID is now a compile error. Closed Sep 26.

This is the rigour needed to remove classes of problems: fix the instance → name the class → detect it → make it impossible to write.

Another example, the currency bug launch fire – members seeing $500K bounties when it should have been $5K, uncaught by all kinds of tests (and human review):

• Aug 4: class – dollars in one place, cents in another.
• Aug 5: detect – a visual snapshot guard on the bounty display.
• Aug 5: make it impossible – one money module (money.ts) and a branded Cents type, so passing a raw dollar amount is now a compile error.
• Aug 5: guardrail on tests themselves – a test value must be one the system actually stores, never one reverse-engineered to make a formula pass.

Observability is good. Self-healing is great. But: the bug that never ships is better. This is the whole point of shift-left. Building in a way that is testable, and actually writing those tests.

The most useful guardrails are architectural. Startups were always about shipping just good enough as fast as possible. But now architectural decisions can support – enable – speed.

As engineers, we use a lot of words to talk about architecture, but at a fundamental level, architecture is about making good decisions easier, on a timeframe you can reason about.

An architectural decision I made at the start was Role-Based Access Control, RBAC. It was an expensive decision in month 1 – and vindicated in month 4.

In the old times that would have been too expensive to justify. We’d had a lot of churn there, and it was somewhere I thought it would be important to be able to add new types, and move things around – one line in a table, and hasPermission() now returns true.

One of my colleagues told me the access model was now stable. What would I have said if I had to justify a detailed roadmap? “I don’t believe you?” Hardly conducive to good working relationships.

As it was, we could deliver the architecture underlying the features without debating the value of it. And when the permissioning model changed – which of course it did – that work paid off, making it less painful.

An architectural decision that surfaced over time was the SQL. It started as a bug: NULL-scan 500s – a NULL in the database reaching code that didn’t expect one.

• Jun 17: fixed the instances, tripwire in CI.
• Jun 18: a ratchet guard stops new hand-written SQL.
• Jun 18-30: moved the handlers’ hand-written SQL onto sqlc-generated queries. That was ~40 PRs – previously an untenable amount of work for something that could be caught in review.

These choices would not have been justifiable in a pre-agentic-development world. But in an agentic development environment, I have seen the impact. Areas of high churn and potential complexity deserve more attention, and more conscious architectural design. The win is seeing fewer problems than you’d expect.

Gates before humans

The shift to orchestration also shifted responsibilities between Claudes. “Dispatch a review agent” was encoded in our process for months. It’s very helpful. But I would still find the odd issue or untracked follow-up. The merge queue was the answer to PR collisions – it serialises merges across concurrent sessions, files the follow-ups, and sends PRs back when they’re not ready.

The next round of reviews I dispatched came back clean. And the next. And the next.

Turns out merge queue Claude, reviewing work it had no part in, was stricter than the Claudes that had done the work. It had value beyond the mechanical sequencing. 

The final quality gate is the release process. It’s easier to have one bigger gate before releasing to users than hope that every PR got the requisite attention. As part of that gate, we use Chromatic visual snapshots to ensure everything that has changed visually is flagged. We also have some extra checks on database changes going to prod because – this is where it all started – data quality is critical to the functioning of the business.

Even if we run a software factory, the first time something meets a human should not be in production.

The evolution of the factory

Here’s the arc of how it went.

Note: this output is two engineers; we had an intern for six weeks in July/August.

We got started in April and May, really hit our groove in June. In July, we were effectively feature complete and spent most of the month preparing for a smooth launch – ensuring our infra was all set up, and doing a ton of work to migrate all the data safely.

August was very reactive. The launch went okay, probably about as well as moving off a no-code platform could have. We did about ten hotfix releases. The real fires were two days at launch – a 500, an SSR crash, and the currency bug. Was it my smoothest launch? No. It was never going to be. It was acceptable. 

In September we came back to having more of a roadmap and delivering bigger features that we couldn’t on the old system, such as a live revenue and metrics reporting page. We also expanded our top of funnel flows for both members and companies, as well as integrations that allow hiring teams to work where they usually work, such as Slack and Ashby. The rest of the output is mostly the second roadmap. In September, fixes were over a third of merged PRs, which includes polish and iteration on things we’d just shipped as well as bugs.

In the end, my answer to the question of what percentage of output should be guardrails is around a third. This was my intuition, and it aligns with the data. Note that this isn’t human time, because so much of this work is quite mechanical.

This doesn’t just show up in the percentages. In September PRs got harder to land – the share landing in ≤2 commits fell from 67% (Aug) to 53% (Sep), and median open time rose from ~1h to 3.5h, partly driven by queue contention. But output was still up – 779 PRs – and September had just one hotfix release.

Operational costs

I had never thought that much about CI before – that was something done by someone who reported to someone who reported to me. But I’ve been spending a lot more time on it, optimising things. We talk about token spend, but those tokens also generate CI minutes. And the more rigour we put onto our CI – where it should live – the more that cost goes up.

In September our CI spend was higher than our AI spend. This was in part thanks to the Max plan (this would not be the case on API metered usage), but also the ability to run the factory during quiet periods – like weekends and overnight – when we don’t need the tokens for everything else, and CI provides the guardrails whilst humans are resting.

I like to split our running costs into base and velocity costs. Base is the cost of running the platform; we’re paying that either way. Velocity costs are a function of output, so they should look like good value against what we’ve delivered for the business. August was our first full month live, and velocity cost (excl. Claude) was about half of base.

The number of guardrails is not the goal; the outcome of reliable software is. Everything needs to add value. I check at least monthly that this is the case.

Running the factory

Scaling a team, you keep coming back to the principle of clarity: be clearer. Clearer about what’s expected, who owns what, what good looks like.

I had a model of engineering management with four layers. None of them went away. What changed is how fast each one moves:

• Mission – the purpose of the team. Still basically the same.
• Strategy – how the team expects to deliver on that purpose. It needs to be more alive. Think in weeks not in quarters, or in quarters if you used to think in years.
• Tactics – the process layer, what makes teams interoperable and observable. It used to be written for humans, took months to land and needed change management; now it’s written for agents, lands the same day and is enforced by machine.
• Execution – getting sh*t done. It went from today or this week to overnight, continuously – the dark factory.

With two developers, our output is high, but we’re still bottlenecked on the bigger feature pieces. Those we single-focus. Generally the way we work is:

• Engineer: one large feature + data work.
• Me: one medium feature, releases, other CTO things (sometimes making slides for VCs and sometimes being the IT department).
• One standalone orchestrator cranking through the backlog.

This means the bigger features still need to be prioritised. They need human attention.

Michelle, our CEO, is non-technical. She only asks for features – that’s her job. The guardrails so she can have those features are my job. I said no to everything for four months, because I knew the velocity win was leaving the no-code platform and I knew scope creep is the killer of the V2.

The best day for me wasn’t the day we shipped. It was the day she asked for something and then mentioned later “oh it’ll be so good when the platform has this…” and I told her it shipped that morning.

When execution gets cheap, doing everything is tempting. But doing everything isn’t a strategy. With the factory model, everything comes back to rigour. What is most valuable, in what order, still needs to be clear.

There’s this dissonance around “go faster” being a strategy. It’s not a strategy. It was never a strategy. Similarly, “just AI it” was never a strategy; it was a forcing function, a blunt instrument, to find the limits. The person who gets paged when those limits are breached, or the person who has to clean it up when it’s wrong, was never going to be on board.

But to say “just AI it” isn’t a strategy is easy and obvious. Replacing that with an actual strategy is much harder. Here’s what my strategy looks like today:

• Every question I have gets an answer. I used to be judicious about this, because every question I had had a human cost. Now I can – and do – ask Claude all my questions until I am satisfied.
• Human communication is AI-augmented, not AI-driven. I spend more time iterating on and reviewing specs for human agreement than I do the PRs that will be produced from them. But the spec is how we ensure we’re building the right thing, well, and it’s the thing that I give to other humans to review and sign off on. It’s critical.
• Everything is in writing in a shared place (skills, CLAUDE.md), and if it can be instrumented effectively, we instrument it.
• We put in checks that allow us to be accountable for work we didn’t produce, and gates where humans explicitly take responsibility – migrations and releases.
• We find non-human ways to surface problems. CI data is a rich source – how long PRs spent open, how many times CI needed to run. Code coverage reports tell us where the gaps are. I’ve built practices (habits and skills, mainly) to check in on this regularly, find the bottlenecks, and figure out how to improve them.

Retrofitting the factory

Building the software factory from the ground up has been fascinating, but the real challenge for the industry is how to retrofit it to existing engineering teams. This is vastly more difficult. 

Managers encode judgement through process and relationships. Relationships build judgement 1:1 – and that’s valuable – but process is how judgement scales. If you want to move at speed with these new tools, you can’t have eight people on the team moving as fast as possible and one person trying to figure out how to make them collectively more impactful. You certainly can’t expect that from the person who has so many direct reports they can’t physically meet with them each week. Every IC needs to learn how to do it for themselves, and then collectively, so the team can move faster together.

ICs aren’t being asked to do this. They’re being asked – told – to go faster. But clarity and good process are how teams go faster. This is the work good managers always did. Now managers need to scale that work earlier, with people who didn’t deliberately sign up for it, and who feel threatened by the idea that the skills they invested in for so long may be less necessary.

Having done both, this is where I would start:

• Prioritise skill building and knowledge sharing for fleet management, because ICs are used to managing their own output and this is a different perspective.
• Rebuild guardrails for the agentic era – historic incident postmortems likely contain ideas of where to start.
• Find non-human ways to surface problems so you can proactively coach ICs on thinking one level up. Fill in platform health gaps, and set up engineering effectiveness checks – like PR open time and CI spend.
• Separate human work from agentic work and be clear about which is which, and why. We don’t get business impact by DDoSing other humans with long, half-thought-through documents. 

Choosing chaos

When the industry was being upended, I chose to go live in the chaos because it was clear to me the chaos was coming for all of us. That coincided with feeling it was time for me to do something different, and honestly, I’ve been having a great time.

How we work is one thing, but I’ll write this one day and it will be different in a week. It’s easier to measure progress than state. This is why I have all the data – it helps me see, amidst the volume, what the impact is, how we’re achieving it, and what is slowing us down.

Scaling and running large orgs, rigour was my job. That piece hasn’t changed. The rigour is what creates progress. Escaping the no-code platform is the unarguable part – there is no way we could have done it without Claude, and also, no way Claude could have done it without us.