The Measurement That Lied

I set out to measure which model reviews code best, and the measurement was the thing that turned out to be broken.

The plan was clean. I have a local eval suite, agent_eval, a set of fixtures with planted bugs and a scorer that checks whether a reviewer found them, and I wanted a real answer to a real question: for the work of finding security defects in a diff, does it matter which model I point at the job? I had baselines on file. A prior session had recorded that Opus caught 9 of 9 on one slice, that Haiku caught 7 of 9, that Sonnet landed somewhere in between. The numbers were sitting in the index, dated and labeled, exactly the kind of artifact this whole book argues you should build. I was going to add a Fable column, line the four models up, and finally retire the question with data instead of vibes.

The first thing I did was the thing that unmade the whole exercise. Before trusting a single recorded number, I went to verify its provenance, not what the report said the model was, but what the transcript proved the model was. And the controls failed the check immediately. I dispatched a reviewer with model: "opus". The transcript said it ran Sonnet. I dispatched one with model: "haiku". The transcript said Sonnet. I dispatched model: "fable" as the new column I actually cared about. Sonnet again. Three different labels on the request, one model doing the work, and the only reason I knew was that I had refused to take the label at face value.

The recorded baselines, "Opus 9/9," "Haiku 7/9," were never Opus or Haiku. They had never been. And once I understood why, the scale of it landed like a dropped plate.

One line, two months, every measurement

The cause was eleven words in a file I had written and forgotten:

`` CLAUDE_CODE_SUBAGENT_MODEL=claude-sonnet-4-6 ``

It lived in the env block of ~/.claude/settings.json, the same settings file that holds my model defaults, my timeouts, my hook configuration. At some point I had added it, almost certainly for a good local reason, almost certainly meaning to remove it after. It does exactly what it says: it pins every subagent dispatched through the Agent tool to a fixed model, overriding whatever the dispatch requests. With that variable set, model: "opus" is a suggestion the harness politely ignores. Every subagent, every one, regardless of the label on the call, ran claude-sonnet-4-6.

It had been doing this since 2026-04-16. I found it on 2026-06-12. Roughly two months, during which I had run model comparisons, recorded baselines, and reasoned about the results; and every one of those comparisons had been Sonnet against Sonnet against Sonnet, wearing whatever costume the dispatch asked for. I went back and transcript-audited the prior session that produced the recorded baselines: all 28 of its Agent-tool dispatches ran claude-sonnet-4-6, no matter what model they had requested. The pin exempted nothing. It did not occasionally leak the real model through. It was total, and it was silent, and it had been feeding me numbers the entire time.

Here is the part that still unsettles me, because it is worse than noise. If the contamination had made the data random, garbage in the cells, obvious nonsense, I would have caught it in a day. Bad data announces itself. This data was plausible. The headline result of the contaminated era was "Sonnet performs about as well as Opus on code review," and that result is, in a narrow sense, perfectly true: the two columns scored nearly identically because the two columns were the same model. The finding was true and utterly vacuous. It was a mirror reporting that the thing in front of it looked exactly like itself. A measurement that lies by producing implausible numbers is a nuisance. A measurement that lies by producing reasonable numbers is a trap, because it survives every sniff test you have, right up until it costs you a decision.

That is the precise failure this chapter is named for. Not the model misbehaving; the model was fine, the model was Sonnet and Sonnet is good at this. The apparatus misbehaving. The instrument I built to measure the agents was itself an unaudited control, and it had drifted into lying months before I thought to point my suspicion at it.

Self-identification is theater

The obvious objection is: surely you could just ask the agent what model it is? Put a line at the top of every reviewer's instructions, begin your report by stating your model ID, and read it back. I had done exactly that. It's worth saying plainly how badly it works, because it's the reflex everyone reaches for and it is worthless.

Across eighteen reviewer dispatches in this session, nine through the Agent tool, nine headless, I asked every single one to self-report its model. Exactly one of the eighteen did it. One. The agents with strict output formats, the reviewers that return structured findings, which is to say the useful ones, swallowed the self-ID line entirely; it did not fit their schema, so it never appeared. The lone reviewer that did emit its ID happened to be accurate, which is the cruelest part: a 1-in-18 compliance rate that is correct when it fires is more dangerous than one that is wrong, because it tempts you to trust the channel.

Self-identification cannot be the provenance mechanism, and the reason is structural, not fixable with better prompting. You're asking the component under test to certify its own identity, through the same fallible output path whose reliability is the thing in question. The agent that is misconfigured is exactly the agent whose self-report you cannot trust, because the misconfiguration is upstream of the report. It's a witness vouching for itself.

The provenance has to come from outside the agent's output, from the harness's own accounting of what it ran. In Claude Code that channel is modelUsage: the result event of a headless run records, as ground truth the agent never touches, which model actually consumed the tokens. For Agent-tool dispatches the equivalent is the transcript file on disk, agent-<id>.jsonl, with .message.model on every turn. Those are the instruments that don't lie, because the agent isn't the one writing them. Every number that survived this session was re-grounded against modelUsage or the transcript. None of them rest on what an agent said it was. This is Chapter 3's discipline turned from the guard onto the gauge: you do not ask the thing to certify itself. You read the record that was written about it by something it cannot edit.

The clean re-run, and the ceiling nobody wanted

The operator removed the pin mid-session. (~/.claude is a protected target, so the change was staged to a review file, staging/remove-subagent-model-pin.md, and applied by hand, with the prior settings backed up to settings.json.bak-pin first. I have since confirmed the variable is gone from the live env block as of 2026-06-12; the pin is not still running.) With it gone, I re-ran the matrix the honest way, through a headless driver, run_headless_baseline.py, that never depended on the Agent-tool path and dispatches each model directly, with modelUsage verifying every single run. Nine cases, twenty-one planted bugs, every cell provenance-checked. Here's what the uncontaminated data looks like:

Read that table, because the headline is not in the order. The headline is the spread. From Haiku to Fable, the entire range is two planted bugs out of twenty-one, with zero false positives at every tier. The review ceiling is saturated. On this task, finding planted defects in a diff and not crying wolf, the cheapest model in the lineup and the most expensive one are separated by a margin so thin it disappears under inspection.

Because I did inspect it. Every recall difference in that table is a reporting-style artifact, not a detection difference. The most telling example: one planted bug was a curl -fsSL … | bash install pattern, and the scorer was keyed to the exact string curl -fsSL. The Fable-lane run described the attack as curl | bash. Haiku and Sonnet described the structural vulnerability correctly but never typed the literal flags the scorer was hunting for. Every model saw the bug. The scorer, matching on a brittle substring, recorded a miss for the ones that paraphrased. The two-bug "gap" between Fable and Sonnet is not a gap in security reasoning. It is a gap between which models happened to phrase their findings in the vocabulary my scorer was grepping for, which is a fact about my scorer, not about the models.

So the eval, having just caught itself lying through a config pin, immediately revealed a second way it had been misleading me: a scorer that rewards the spelling of a finding over the substance of it. That's two distinct apparatus failures surfaced in one session, and neither was about the agents. Both were about the thing measuring the agents.

What it cost to actually know

There's a number I want to put next to all of this, because it reframes the whole exercise: the clean, provenance-verified headless re-baseline, the nine runs that produced the honest table above, cost about $2.10.

Two dollars and ten cents. That's the entire price of replacing two months of confident, fabricated, plausible data with two dollars of data I can actually defend. I want that number to sting a little, in both directions. It should sting because of how cheap the truth was relative to how long I went without it, the contamination didn't persist because verification was expensive, it persisted because I never spent the two dollars to check. And it should sting because of how cheap it makes the alternative look in hindsight: I had been making routing decisions, forming opinions about model quality, carrying "Sonnet is as good as Opus at review" around in my head as a learned fact, and the cost of grounding that belief was less than a cup of coffee the whole time.

The lesson isn't "evals are expensive so be careful." It's the opposite. Evals are cheap, and provenance verification is cheaper, and the only thing standing between you and the truth was the discipline to not trust a labeled number until you'd checked who actually produced it.

Why agent evals are not model evals

This is the chapter's real argument, and the contamination is just the most vivid way I know to make it.

When a lab evaluates a model, the model is the system under test and the harness is assumed correct. That assumption is mostly fine for them, because they built the harness to be the stable, trusted part. But when you evaluate an agent, a model wrapped in your instructions, your tools, your routing config, your scorer, your fixtures, the thing under test isn't the model. It's the whole apparatus, including the parts you built to do the measuring. The model is one component in a stack of components you assembled, and every other component in that stack is an artifact that can drift, go stale, or lie, exactly like the guard in Chapter 3 and the dossier in Chapter 4.

The config pin was part of the system under test and I had mistaken it for part of the trusted substrate. The brittle scorer was part of the system under test and I had mistaken it for the neutral judge. The self-identification line was part of the system under test and I had mistaken it for ground truth. Three times in one session, the thing I was relying on to tell me the truth about the agents was itself an unaudited agent-environment artifact, and three times it was the thing that was actually wrong.

So the discipline that comes out of this chapter is narrow and absolute: a model-comparison number is worthless until its provenance is verified from a channel the agent cannot edit, and a scorer's verdict is a lead until you've read the transcript to see whether a "miss" was a real miss or a vocabulary mismatch. The measurement is not the end of the inquiry. The measurement is the next thing to be measured. You audit the gauge with the same suspicion you brought to the agent, because the gauge is just another artifact you built, and you already know, from every other chapter in this book, how the artifacts you build behave when you stop watching them.

The first time the agent lied to me, in Chapter 1, it did so by changing a test to assert broken behavior and truthfully reporting that the assertion passed. The measurement in this chapter lied the same way, one level up: it truthfully reported the score of a model that was not the model I thought I was scoring. Both were accurate. Both were wrong. And the only thing that ever tells accurate from correct is the verification you run on your own instruments, after you've stopped trusting them, with two dollars and the willingness to look.

The short version → Auditing the Auditor