Why the bottleneck on AI isn't intelligence, it's trust

Our PR volume has climbed steadily this year. What hasn’t grown is the time I have to actually read any of them. A growing share get written by an AI assistant, and a growing share of the review comments on them come from another AI, tuned to catch the obvious stuff fast. By the time a PR reaches me, it’s already been generated by a model and pre-approved by a model. My job, more and more, is deciding whether to trust that chain, not reading the diff line by line.

That’s not a complaint about anyone being lazy, myself included. There simply isn’t enough senior attention to go around at the rate code gets produced now. I didn’t have a name for that gap until I read Some Simple Economics of AGI by Christian Catalini, Xiang Hui and Jane Wu. Strip away the equations and it comes down to one idea: the cost to automate is falling faster than the cost to verify. Everything I’ve been feeling on my team follows from that one gap.

The two costs racing each other

Picture two costs for any task, plotted over time.

The cost to automate depends on compute and how much knowledge already exists about the task. Both keep growing, so this cost keeps falling, for almost everything.

The cost to verify depends on something much slower: how much experience a human has, and how long it takes for a mistake to actually show up. A typo in a comment shows up instantly. A subtly wrong assumption baked into a service boundary might not show up for two quarters. That cost barely moves, because my time and the team’s experience don’t scale the way compute does.

The paper calls the gap between those two lines the measurability gap. It’s the space where an agent can act cheaply, but nobody can afford to check whether it acted correctly. Reading it, I recognized my own review queue.

Every task lands in one of four zones

Once you have those two costs, you can sort any task into one of four boxes: cheap or expensive to automate, affordable or unaffordable to verify.

A small, well-scoped change, a login form, a one-function bug fix, sits in the safe industrial zone: cheap to generate, cheap for me to check in a minute. A sprawling change that touches a service boundary, three files deep in business logic, whose bugs won’t surface until two sprints from now, sits in the runaway risk zone. That’s where more of our merged PRs live every month, and it’s exactly where the incentive is to ship anyway: the upside is immediate, the cost of being wrong is deferred to someone else’s on-call shift.

Why the risk keeps growing on a team like mine

The paper points to three mechanisms, and I’ve watched all three happen at work.

1. The missing junior loop. Juniors used to build expertise by doing the routine, low-stakes work: the CRUD endpoint, the plumbing, the tedious refactor. That’s exactly the work we now hand to AI first. Automate it away, and you remove the training ground for the people who’d eventually be qualified to review AI output themselves.
2. The codifier’s curse. Every review comment I leave on an AI-authored PR is a correction the tooling can learn from. I’m not just reviewing today’s code, I’m generating the training signal that makes the next model need less of my review. Verifying it now quietly trains the system to need me less, and trains fewer people like me at all.
3. Alignment drift. When the second AI is grading the first AI’s PR against “tests pass, lints clean,” it optimizes for exactly that, not for the invariant we actually care about. Nobody’s watching closely enough to catch the drift before it ships. That’s Goodhart’s law, running at machine speed, inside my own repo.

What happens if nobody catches it

Unverified code doesn’t just disappear. The paper calls it a Trojan Horse: work that looks completely fine on every dashboard while hiding a failure that hasn’t surfaced yet. A PR that passes CI, passes lint, passes the AI reviewer, and still ships a wrong assumption that only shows up in production eight months later, once enough of these accumulate, you get what the paper calls a Hollow Economy: velocity metrics look great while the actual reliability of what you’ve shipped quietly erodes underneath them.

What I’m actually doing about it

The paper’s answer isn’t to slow AI down, it’s to treat verification as infrastructure worth investing in directly: tooling that makes a model’s output easier to inspect, deliberate practice that keeps rebuilding the expertise our juniors used to get for free, and systems that fail safely when nobody’s watching closely enough.

On my team, that means being honest about what “senior engineer” is actually being paid to protect right now. It’s not throughput. I still don’t have a clean answer for how to structure that, given how thin review time already is. But I’ve stopped treating an AI-approved PR as a reviewed one, and I’d rather ship less this quarter than find out which of our PRs was the Trojan Horse.

If you’re reviewing more AI-written code than you used to, I’d guess you’ve felt this gap too, even if nobody on your team has said it out loud yet.