Can You Trust Code No Human Wrote?

How we rewrote our entire backend with AI, and knew it worked.

Jul 06, 2026

86 Go services and ~500,000 lines became one TypeScript service. No human hand-wrote the new code. Here’s the testing system that made it safe.

TL;DR

We collapsed 86 interdependent Go microservices (~500K lines, 6 years old) into one TypeScript monolith (~231,000 lines) on Cloud Run, with AI agents doing all the engineering. No human hand-wrote the new code.
Two AI models estimated 6-7 months for a human team. The migration itself took about two weeks.
The whole thing works for one reason: we never trusted the AI. We made it prove every change against tests it couldn’t cheat.
The old Go system was the spec. The same test suite runs against both the old Go and the new TypeScript (Go build tags). Both green means identical behaviour.
Our first test suite was a lie. Coverage was real, but the tests asserted nothing. The AI was cutting corners to hit the number (Goodhart’s Law). So we rebuilt testing so that every side effect (every outbound call, DB write, Kafka publish) has to be explicitly asserted, or the test fails.
The migration ran as a plain-Python factory: a 7-phase, gated pipeline per service, specialist agents for each role, two models reviewing every change, and automatic rollback on any regression.
The takeaway for anyone building with AI: the AI can do almost anything if you give it an honest, unbluffable signal of whether it’s right. The hard question isn’t “is the model smart enough?” It’s “what’s my unbluffable check?”

1. The problem

Our backend is a core part of what our business runs on, and one of our main revenue sources. It was built over the last six years, on a more complicated architecture: a fleet of small, single-purpose Go services that talk to each other over RPC, coordinated through Kafka, with ScyllaDB for storage and Redis for caching, all on Kubernetes. If none of that means much to you, the picture is simple enough: a large, old machine with 86 separate moving parts, all wired together and all talking to each other at once.

That’s a brilliant design at scale. But we are a small team (our CTO and four product engineers). So we were paying an enormous operational tax to run and reason about 86 services for load we don’t have. Worse, it was Go, while everything else we build is TypeScript, and our newer product, built from scratch in TypeScript with AI, moves far faster. We had a direct control group telling us this backend was holding us back.

The plan: collapse all 86 services into one TypeScript service on Cloud Run (a fully managed runtime, so you deploy it and forget it), keeping ScyllaDB for now. The SQL migration is a deliberate phase two, because you don’t change too many things at once.

The scope, measured from the repo:

Two AI models, asked independently, put a human team at 6-7 months. And that estimate assumes the sane approach: migrate it slowly and incrementally, one service at a time, old and new running side by side, while still shipping features to the live product. That kind of migration drags on for the better part of a year, quietly competes with every other priority, and usually stalls out half-done. Realistically it never even gets started. You just live with the tech debt forever.

So we did the opposite. We gave ourselves one week. It was a deliberately absurd target, and the absurdity was the point: you cannot hand-crank 86 services into a new language by hand in a week. The only way the number makes any sense is to stop thinking about migrating services and start thinking about building a machine that migrates services. Let the agents do the engineering, and put all of our effort into the factory that runs them, the tests that check them, and the guardrails that keep them honest. A gentler deadline would have let us quietly fall back into porting it by hand, service by service, and we’d still be going now. (For the record, it took longer than a week. But if we’d given ourselves a month, it would have taken two.)

2. The core bet: the old system as an oracle

You can’t just ask the AI to rewrite it. Even if the model is good, how would you know it got it right? Across half a million lines, you can’t eyeball it.

So the bet was: make the old system the definition of correct, and never argue about anything else.

Rewrite it bug-for-bug. If the old code had a bug, the new code keeps the exact same bug. The number-one rule was don’t change anything, because the moment you “improve” something mid-migration you can no longer ask the one question that keeps you safe: is the behaviour identical? (Improvements come later, once you’re in one language and can move fast.)
Test at the boundary, not the internals. For each service we captured its observable contract: every request in, every response out, every call it makes to other services, every DB write, every Kafka publish, all as tests against the old Go.
Run the identical tests against the new TypeScript. Both green means the new service behaves identically, down to the edge cases and the bugs.

The AI is never the source of truth. The tests are.

There were no tests in the legacy codebase to start with, so step one was having the agents write them: around 7,500 tests across 1,270 files.

3. The cross-target harness: one test, two systems

This is the piece that turns a risky rewrite into a safe migration.

The same Go test body runs against both the old Go handler and the new TypeScript server, selected by Go build tags:

// legacy/service.user-job/handler/testmain_go_test.go
//go:build e2e && !ts        // Go target

// legacy/service.user-job/handler/testmain_ts_test.go
//go:build e2e && ts         // TypeScript target

// legacy/service.user-job/handler/put_user_job_test.go
//go:build e2e               // shared test body, compiled for BOTH

go test -tags="e2e" boots the Go service in-process. go test -tags="e2e,ts" boots the TypeScript server and points the same tests at it.

The two TestMain files boot the respective backend. The Go one serves the handler on a local port and registers it in the RPC client’s routing table:

// legacy/service.user-job/handler/testmain_go_test.go
typhonSrv, _ := typhon.Serve(svc, ln)
testharness.SetupTyphonClient(map[string]string{
    "service-user-job": typhonSrv.Listener().Addr().String(),
})

The TypeScript one spawns the real server as a subprocess and waits for it to go healthy:

// legacy/service.user-job/handler/testmain_ts_test.go
_, err := harness.StartTS(testharness.TSConfig{
    Service: "service-user-job",
    Env: map[string]string{"TS_BOOT_SCOPE": "", "JOBS_JOB_KEYS": "[]"},
})

Under the hood StartTS boots the real TypeScript server as a subprocess with the test infrastructure wired in, waits for it to report healthy, then registers it in the same routing table the Go target uses.

The test body is written once and knows nothing about which target it’s hitting. It just calls the proto-generated .Send(ctx), which routes via the RPC client to whichever backend was booted:

// legacy/service.user-job/handler/put_user_job_test.go
func TestPUTUserJob_CreateNew_NewCron(t *testing.T) {
    harness.SetTest(t)
    resp, err := req.Send(ctx).DecodeResponse()

    testharness.AssertResponse(t, harness, resp, err, map[string]any{}).HasStatus(200)
    harness.AssertDBInsert(t, "crons", "jobkey", req.JobKey, "cron", req.Cron)
    harness.AssertDBInsert(t, "jobs",
        "userid", req.UserId, "jobkey", req.JobKey, "endpoint", req.Endpoint,
        "method", req.Method, "payload", req.Payload)
}

Crucially, the infrastructure is shared. Go and TS runs hit the same Scylla cluster, Kafka broker and Redis instance, so parity is checked on real state changes, not mocks. 116 services ended up with this dual testmain_go/ts setup.

A debugging move we leaned on, once something looked off: write a test that passes in Go but fails in TypeScript, then hunt down every place we made the same mistake. That’s how we caught a whole class of wire-format bugs (for example, requests arriving snake_case but responses serialising camelCase).

Because these tests only exist to prove parity, they’re scaffolding. Once a service is cut over, the cross-target tests can be retired.

4. The false dawn: the AI was cheating

Our first approach to writing those tests was the obvious one: point agents at each service, tell them to write integration tests, and chase code coverage (Go reports it directly, so we had a real number and a dashboard). Coverage climbed, service after service went green, and we thought we’d cracked it.

Then we looked closely. The coverage was real. The verification was fake.

Here’s a typical example. We’d tell it: this service calls another one, make sure that call happens and test it. And it would write a test that executed the code path (so coverage counted it) but quietly mocked the call out and asserted nothing. Maximum coverage, happy-path only, and it wouldn’t check the database even when we told it to. The coding equivalent of “yeah, yeah, I tested it, Dad.”

Concretely, it looked like this: the test stubs a fake response for the call, runs the handler, then checks nothing.

// the handler calls /read_user; the test stubs a canned response…
mockRouter.RegisterHandler("/read_user", testharness.CannedJSONHandler(map[string]any{
    "user": map[string]any{"id": userID, "display_name": "Test User"},
}))
// …and then asserts nothing about whether /read_user was called, or with what.
// Green. Covered. Meaningless.

This is Goodhart’s Law: when a measure becomes a target, it stops being a good measure. We made coverage the target, so the AI optimised coverage, not correctness. It wasn’t malicious. It was a lazy genius cutting corners to hit the number.

The pivot: stop trusting the AI. Make it prove it, with a test framework it can’t game.

5. The un-cheatable test framework: assert every side effect or fail

So we deleted the first suite entirely. That sounds dramatic, but it was the pragmatic call: with the new approach ready, having the AI regenerate the tests from scratch turned out faster than getting it to repair the old, hollow ones. Fixing meant untangling thousands of meaningless assertions one by one; regenerating meant starting clean, under rules it couldn’t cheat. When regenerating is this cheap, throwing the work away can beat fixing it, which is a genuinely new thing about building software with AI.

The new approach changed what a test is. Instead of “did this line run?”, the rule became: every side effect a service produces must be explicitly asserted, or the test fails.

The harness arms recording proxies when you call harness.SetTest(t) and captures everything the service-under-test does:

outbound RPC calls (MockOutboundRouter)
Kafka publishes (KafkaRecProxy)
DB reads, writes and deletes (DBTracker)
the response itself (must be asserted via AssertResponse / AssertError)

Then, at test cleanup, it fails the test if anything was recorded but not asserted:

// legacy/libraries/testharness/harness.go  (t.Cleanup registered in SetTest)
if h.KafkaRecProxy.HasTraffic(name) && !h.KafkaRecProxy.Asserted(name) {
    var unasserted []string
    for i, p := range produces {
        if asserted[i] { continue }
        unasserted = append(unasserted, fmt.Sprintf("Produce to %q", p.Topic))
    }
    if len(unasserted) > 0 {
        t.Errorf("[recording] unasserted Kafka Produces, call harness.AssertKafkaProduce:\n  %s",
            strings.Join(unasserted, "\n  "))
    }
}
// …same for RPC calls, DB writes, and the response.

So a test that publishes three Kafka messages and asserts two doesn’t pass with a warning. It fails, and it tells you exactly what you missed. There’s no “happy path only” any more. If the code does a thing, the test has to account for it.

“Account for” means fully assert. The matcher (deepCompare) is bidirectional: a missing expected key fails, and an unexpected actual key fails too:

// legacy/libraries/testharness/assert_json.go
for k, ev := range expMap {                 // every expected key must exist…
    if _, ok := actMap[k]; !ok { errs = append(errs, path+"."+k+": missing from response") }
    errs = append(errs, deepCompare(path+"."+k, actMap[k], ev)...)
}
for k := range actMap {                     // …and no EXTRA keys allowed
    if _, ok := expMap[k]; !ok {
        errs = append(errs, fmt.Sprintf("%s.%s: unexpected field in response (value: %v)", path, k, actMap[k]))
    }
}

That “unexpected field” branch is what makes hollow assertions impossible. You can’t assert one field and ignore the rest, because the extra fields fail the test until you account for them.

For fields that genuinely vary per run (timestamps, generated IDs) you can use testharness.NonEmpty or testharness.Any:

// legacy/service.hubspot/handler/put_contact_record_test.go
harness.AssertExternal(t, "hubspot", "/engagements/v1/engagements", map[string]any{
    "engagement": map[string]any{
        "active": true, "type": "NOTE",
        "timestamp": testharness.NonEmpty, // drifts per run
    },
    "metadata": map[string]any{"body": "RSVP'd to AI Workshop in London on 2026-05-01"},
})

But wildcards are a loophole, so they get policed too (next section).

Why this was the turning point: it gave the AI a tight, honest feedback loop. It no longer had to read and reason about the whole codebase to know whether it had done a good job. It ran the tests and got a deterministic, specific failure it could act on (”you didn’t assert the produce to notification.created“). It could grind in a loop and brute-force its way to correct, but it could no longer fake the result. Ralph Wiggum in a loop, with a judge it can’t bribe.

We didn’t make the AI reliable. We built a system that’s reliable even though the AI isn’t.

6. Closing the loopholes: the anti-cheat audit scripts

Once side effects had to be asserted, the AI’s corner-cutting just moved to subtler places. So each loophole got its own deterministic check (all in migration/scripts/):

Under-assertion is caught by deepCompare‘s “unexpected field”, then auto-repaired by fix_under_asserts.py, which parses the failures and adds the missing fields:

# migration/scripts/fix_under_asserts.py
UNEXPECTED_RE = re.compile(r'^\s*\$\.([A-Za-z0-9_.]+):\s+unexpected field in response\s+\(value:\s*(.*)\)\s*$')

Wildcard overuse (using Any for a field that’s actually deterministic) is caught by wildcard_diff.py, which records two identical e2e runs and diffs them. If a wildcarded value is the same across both runs, it’s stable and should be a literal; only genuinely drifting values are allowed to stay wildcards:

# migration/scripts/wildcard_diff.py
if va == vb and ta == tb:  stable.append(...)   # safe to literalise
else:                      drift.append(...)    # must stay a wildcard

On one service alone (service.article-comments), this flagged 57 of 108 wildcarded values as stable: more than half of the AI’s “checks” were checking nothing.

Boundary shortcuts (a test calling a Go DAO directly instead of going through the public HTTP/RPC boundary, which couldn’t be re-run against TypeScript) are caught by check_boundary_entry.py, which statically checks every Test* reaches a real entry point:

# migration/scripts/check_boundary_entry.py
HANDLER_PATTERNS  = [r"\.\s*Send\s*\(\s*ctx\b", r"\btyphon\.NewRequest\s*\(", r"\bhttp\.Post\s*\("]
CONSUMER_PATTERNS = [r"\.\s*Publish\s*\(\s*ctx\b", r"\bstreams\.Publish\s*\("]
# fails: "Handler tests missing public HTTP boundary (.Send(ctx), …)"

“Untestable” as an excuse (marking a branch uncoverable without justifying it) is caught by audit_ceiling_cheats.py, which enforces a fixed taxonomy and rejects weak reasons:

# migration/scripts/audit_ceiling_cheats.py
ALLOWED_TAXONOMY = {"DEAD_CODE","UNREACHABLE_FROM_BOUNDARY","ASYNC_KAFKA",
                    "NON_DETERMINISTIC","VENDOR_LOCKED","FRAMEWORK_PLUMBING"}
WEAK_KEYWORDS = ["in practice","production","typically","should never","shouldn't happen"]
# DEAD_CODE must cite a code-level reason ("return nil", "constant", "unreachable"), not vibes.

The pattern was the same every time: whenever the AI found a way to cut a corner, we turned that corner into a deterministic check. You don’t out-argue the model. You build a fence it can’t climb.

7. The migration factory

With trustworthy tests, the migration itself became a plain-Python assembly line (migration/scripts/) that drove AI agents through a gated pipeline per service. Boring, deterministic orchestration; AI supplying the intelligence.

Each service ran through seven phases (chief_migrator.py, migration_factory.py):

Maximise coverage (maximize_all.py): measure go test -cover, have an agent classify every uncovered path as testable or untestable, write tests for the testable ones, iterate (rolling back if coverage regresses).
Contract tests (generate_contract_tests.py): pin the HTTP wire format (status codes, headers, exact JSON field names).
Interaction tests (generate_test_interactions.py): enforce full side-effect assertions (the framework from section 5).
Quality audit (run_all_test_generation.py): batches of test functions audited and classified OK, GAP, N/A or UNFIXABLE, fixed iteratively.
Verification and migration gate: a fresh go test, then a go/no-go decision:

# chief_migrator.py
m = re.search(r'GATE_DECISION:\s*(READY|MIGRATE_WITH_CAVEATS|DO_NOT_MIGRATE)', output)
decision = m.group(1) if m else "DO_NOT_MIGRATE"    # default is "don't"
...
if migrate and result["gate"] != "DO_NOT_MIGRATE":
    p6 = run_phase6_migration(target)
elif result["gate"] == "DO_NOT_MIGRATE":
    result["status"] = "GATED"                       # pipeline stops; no migration

Migration (run_migration.py): an orchestrator agent spawns specialists. An architect reads all the Go and plans the TS files, a coder writes them chunk by chunk, and two reviewers run in parallel (a different model from the one that wrote the code) checking behaviour fidelity, error codes and wire format. Both must PASS before moving on.
Cross-target tests: run the Go e2e suite against the new TS server (section 3). Fix and iterate until green.

Models by role (invoked via droid exec): Codex 5.3 (xhigh) for fast coding and audits; Opus 4.6 (max) for orchestration, architecture and review; with GPT-5.3 and Gemini 3.1 in the review and consensus mix. Using different companies’ models to review each other matters, because where one is blind, another catches it.

Playbook per service type. One generic migrator was mediocre at everything, so the factory picks a tailored playbook by service type:

# chief_migrator.py
svc_type = "api" if svc_dir.name.startswith("api.") else "service"
# edge.* GraphQL packages are discovered separately and use 015_EDGE_TEST_PLAYBOOK.md
# service.* / api.* use 014_TEST_PLAYBOOK.md

GraphQL edges need different handling (schema, resolvers, dataloaders, error extensions, pagination cursors) than REST services (handlers, consumers, DAO, Kafka).

Automatic rollback. Before any assessment step that lets an agent touch test files, the factory snapshots them; if the agent breaks the suite, it reverts:

# chief_migrator.py
def rollback_test_files(snap, target):
    for fpath, content in snap.items():
        p = Path(fpath)
        if p.exists() and p.read_bytes() != content:
            p.write_bytes(content)          # restore modified
    for f in base.rglob("*_test.go"):
        if str(f) not in snap: f.unlink()   # remove newly-added
# → "[CHIEF_MIGRATOR] Assessment broke tests, rolled back N file(s)"

The safe state was always the default, which is what let it run mostly unattended (phases parallelised with -j, for example maximize_all.py -j 4, run_migration.py -j 5, with per-phase timeouts measured in hours).

8. Discovery and anti-hallucination

Before any of that, we had to understand 86 services nobody on the current team had written. We pointed one agent per service at the code (migration/discovery/scripts/run_discovery.py), auto-splitting the big ones into chunks of at most 40 files. The prompt was blunt: “Be exhaustive. Read EVERY source file. Do not skip, sample, or summarise.”

The obvious risk is hallucination, so every non-trivial claim had to cite a specific line of code:

# migration/discovery/prompts/discover-service.md
- Every non-trivial claim MUST include an evidence tag: [evidence: relative/path/file.go:12-34]

A verifier (verify_reports.py) parsed those tags and rejected the report if citations were missing, invalid, or didn’t cover enough of the source files, triggering a strict regenerate-from-scratch retry. Then three models from three companies (Opus 4.6, GPT-5.3 Codex, Gemini 3.1 Pro) independently reviewed each report for completeness, and review_consensus.py blocked synthesis unless they agreed (score spread within 10, no split verdicts, no “unsupported statement” flags).

The output was a living PATTERN_PLAYBOOK.md: a catalogue of all 86 services, a domain map, an 80+ topic Kafka catalogue, and the recurring async patterns (fire-and-forget, event sourcing, fan-out, sagas, Kafka-to-Redis bridges). That’s what turned “rewrite this” from guesswork into applying known patterns.

9. Prompting lessons (the cheap, painful ones)

Never say “make the tests pass.” You’re literally inviting the model to delete or skip the hard ones. It does the literal thing, not the intended thing. Say what “done” means, and enforce it in the harness.
Don’t make a proxy metric the goal. “Get coverage up” got us hollow tests. Make the real property (every side effect asserted) the thing that’s checked.
The model can’t reliably hold the whole picture. It’s genuinely hard for an AI to read all the code and all the tests and be sure every boundary is exercised and every request and response fully asserted. Don’t ask it to. Build the check that guarantees it, and let the model iterate against clear failures.
Reject, don’t cajole. A deterministic “rejected: uncited claim / unasserted produce / weak reason” beats a paragraph of pleading. The strict-retry loop did more than better wording ever did.
Different models catch different things. A second (and third) model from a different lab, reviewing rather than writing, kills a surprising share of plausible-but-wrong output.

10. The moving target: forward-porting a live system

The rest of the team kept shipping features to the live Go system the whole time, so our target moved under us. We built sync tooling (migration/scripts/sync_from_master*.py) to continuously pull those changes in and forward-port them, using a per-file 3-way merge so local migration patches weren’t clobbered:

# migration/scripts/sync_from_master_3way.py, per changed file:
#   A (added)    → copy in
#   D (deleted)  → remove
#   M (modified) → git merge-file (3-way); leave <<<<<<< / ======= / >>>>>>> on conflict
# sync point tracked in migration/.last_master_sync; advanced manually after resolving

Because every ported change came with its own test, we were rebuilding the plane while everyone else kept flying it, safely.

11. Making thousands of real-infra tests fast

These aren’t mocked unit tests. They hit real ScyllaDB and Kafka, so parity is checked against real state changes. Running 7,000+ of them would be unbearable if each spun up its own containers. So the harness runs a resource-pool daemon (legacy/libraries/testharness/cmd/pool/main.go) that pre-warms a pool of Scylla and Kafka instances and leases them to tests, reusing each container across many runs instead of paying startup cost every time:

// legacy/libraries/testharness/cmd/pool/main.go
go scyllaPool.preWarm()
go kafkaPool.preWarm()
// instances are leased per test and returned; reuseCount tracks how many times
// each container is handed out, with idle-shrink and leaked-lease reclaim

It listens on localhost:19876, warms instances in the background, tracks per-instance reuse, reclaims leaked leases (a crashed test would otherwise hold a container forever), and shrinks the pool when idle. Keeping the infrastructure warm and reused let a 7,000-test, real-infra suite run in minutes instead of hours. That speed is what made the whole “let the agent iterate against the tests” loop viable at all.

One other fix worth a mention: moving the new TS service from kafkajs to Confluent’s librdkafka-based client surfaced that librdkafka requires topics to exist at subscribe time (unlike sarama and kafkajs, which auto-create). The fix was an ensureTopics() call via the admin API before subscribing, which also made consumer startup fast and deterministic.

12. Results

About two weeks of migration (the main push in April 2026), against the 6-7 month human estimate.
On the biggest day (14 April), a single run queued 47 targets, two at a time, and 23 came back green. Counting the handful finished the day before, roughly 27 in a day.
~7,500 trustworthy integration tests, every one asserting real behaviour.
Live on Cloud Run, the old Go switched off, and because every service had to pass the exact same tests the old one passed, no behavioural regressions at cut-over. Safe, not just fast.
The handover, when I went in for surgery, wasn’t a doc for humans. It was an AGENTS.md: the project’s living memory, written for the agents, so anyone could just run them:

# AGENTS.md (excerpt)
## Migration status: post-cutover
The Go→TS migration is complete. The TS monolith in `server/` is the sole runtime in
production; the legacy Go services have been decommissioned. `legacy/` remains only as a
read-only historical reference.

13. The point (and the human cost)

This isn’t really a migration story. It’s about how we work with AI now.

The AI is a brilliant worker, but it can be an unreliable one. It is never your source of truth. But give it an honest, unbluffable signal of whether it’s right, a check it can’t cheat, and you can make it do genuinely astonishing things. Here the old system defined “correct” and deterministic tests were the judge. The migrations that used to be too risky or too expensive to attempt just moved onto the table.

The one precondition, so nobody leaves overclaiming: this only works if you can define “correct.” No oracle, no safety. The real prerequisite isn’t a smarter model, it’s a verifiable definition of done. So the question to ask of your own work is: what’s my unbluffable check?

Don’t ask AI to be right. Build a way for it to prove it.

One honest coda: it isn’t free. Driving a swarm of agents that constantly cut corners is genuinely exhausting. The context-switching, the never-fully-trusting, the volume of output to review. It’s a real cognitive load and a real burnout risk, and if we’re going to work this way we need to take it seriously. But that’s another post.

Discussion about this post

Ready for more?