Evals are the new code review

Code review exists because writing code is where mistakes enter. So we put a gate right there: another set of eyes on the diff before it ships.

When agents write a growing share of the code, that gate stops catching what breaks. The failure is rarely in the syntax anymore. It's in the behaviour. The model does something plausible and wrong, or right in the demo and wrong on the rare inputs, and a person reading the diff line by line won't see it, because the diff looks fine. The problem lives in the output, not the source.

Code review exists because writing code is where mistakes enter.

So the checkpoint moves. The question shifts from "is this code correct" to "is this behaviour correct, across the inputs that matter, reliably enough to ship." That's an eval: a set of cases you run against the output, with a pass mark you set in advance. A team still gating on human code review while agents write the code is checking the building materials and shipping the behaviour unchecked.

Evals are to AI-built products what code review was to hand-written ones. Same discipline, different object. You define what good looks like. You build the cases that test it. You run them before release. You don't ship past a failing gate. Teams that treat this as core engineering practice ship AI that holds up. Teams that treat it as a nice-to-have ship things that work until they don't, in front of a customer.

The part most teams get wrong is who does the checking. Checking is a routing decision now, and the rule is cheapest reliable check first. Rules go first: is the output the right shape, the right format, free of banned terms, inside the thresholds. Anything a rule can settle, a rule should settle. What a rule can't settle, hand to a model with clear guidance. That's cheap enough to run on every change and consistent enough to catch behaviour sliding away from what you asked for. A person comes in last, and only for calls that set a standard. Route it the other way and you pay a person's rate for work a rule would have covered, and you still miss the rare inputs, because a person can't read ten thousand outputs and a rule can.

That's also the line agents don't cross. An agent can do the read, produce the draft, and run the check. It can tell you six outputs look off and show you why. It doesn't get to decide the threshold, and it doesn't get to decide that a failing case is acceptable this once. One decision, one owner, and the owner is a person. The check is delegated. The standard isn't.

The tell that you've under-invested here is that confidence about quality comes from how it felt. It looked good when someone tried it. That was the state of code quality before code review, and we didn't accept it there. A feature isn't done because it demoed well. It's done when it passes the gate that says it does what it's supposed to on the cases you care about, and when you can say which parts of that are measured and which are still a bet.

This is where taste becomes infrastructure. Deciding what to evaluate, and what threshold counts as good enough, is a judgment call. It is one of the most valuable calls a builder makes now. The eval suite is where your standard for quality stops being an opinion and becomes something the system enforces on every release.