# The Measurement That Lied

I set out to measure which model reviews code best, and the measurement was the thing that turned out to be
broken.

The plan was clean. I have a local eval suite, `agent_eval`, a set of fixtures with planted bugs and a scorer
that checks whether a reviewer found them, and I wanted a real answer to a real question: for the work of
finding security defects in a diff, does it matter which model I point at the job? I had baselines on file. A
prior session had recorded that Opus caught 9 of 9 on one slice, that Haiku caught 7 of 9, that Sonnet landed
somewhere in between. The numbers were sitting in the index, dated and labeled, exactly the kind of artifact
this whole book argues you should build. I was going to add a Fable column, line the four models up, and
finally retire the question with data instead of vibes.

The first thing I did was the thing that unmade the whole exercise. Before trusting a single recorded number,
I went to verify its *provenance*, not what the report said the model was, but what the transcript proved the
model was. And the controls failed the check immediately. I dispatched a reviewer with `model: "opus"`. The
transcript said it ran Sonnet. I dispatched one with `model: "haiku"`. The transcript said Sonnet. I dispatched
`model: "fable"` as the new column I actually cared about. Sonnet again. Three different labels on the request,
one model doing the work, and the only reason I knew was that I had refused to take the label at face value.

The recorded baselines, "Opus 9/9," "Haiku 7/9," were never Opus or Haiku. They had never been. And once I
understood *why*, the scale of it landed like a dropped plate.

## One line, two months, every measurement

The cause was eleven words in a file I had written and forgotten:

```
CLAUDE_CODE_SUBAGENT_MODEL=claude-sonnet-4-6
```

It lived in the `env` block of `~/.claude/settings.json`, the same settings file that holds my model
defaults, my timeouts, my hook configuration. At some point I had added it, almost certainly for a good
local reason, almost certainly meaning to remove it after. It does exactly what it says: it pins every
subagent dispatched through the Agent tool to a fixed model, overriding whatever the dispatch requests. With
that variable set, `model: "opus"` is a suggestion the harness politely ignores. Every subagent, every one,
regardless of the label on the call, ran `claude-sonnet-4-6`.

It had been doing this since **2026-04-16.** I found it on **2026-06-12.** Roughly two months, during which I
had run model comparisons, recorded baselines, and reasoned about the results; and every one of those
comparisons had been Sonnet against Sonnet against Sonnet, wearing whatever costume the dispatch asked for. I
went back and transcript-audited the prior session that produced the recorded baselines: **all 28 of its
Agent-tool dispatches ran `claude-sonnet-4-6`,** no matter what model they had requested. The pin exempted
nothing. It did not occasionally leak the real model through. It was total, and it was silent, and it had been
feeding me numbers the entire time.

Here is the part that still unsettles me, because it is worse than noise. If the contamination had made the
data *random*, garbage in the cells, obvious nonsense, I would have caught it in a day. Bad data announces
itself. This data was *plausible*. The headline result of the contaminated era was "Sonnet performs about as
well as Opus on code review," and that result is, in a narrow sense, perfectly true: the two columns scored
nearly identically because the two columns were the same model. The finding was true and utterly vacuous. It
was a mirror reporting that the thing in front of it looked exactly like itself. A measurement that lies by
producing implausible numbers is a nuisance. A measurement that lies by producing *reasonable* numbers is a
trap, because it survives every sniff test you have, right up until it costs you a decision.

That is the precise failure this chapter is named for. Not the model misbehaving; the model was fine, the
model was Sonnet and Sonnet is good at this. The *apparatus* misbehaving. The instrument I built to measure
the agents was itself an unaudited control, and it had drifted into lying months before I thought to point my
suspicion at it.

## Self-identification is theater

The obvious objection is: surely you could just ask the agent what model it is? Put a line at the top of every
reviewer's instructions, *begin your report by stating your model ID*, and read it back. I had done exactly
that. It's worth saying plainly how badly it works, because it's the reflex everyone reaches for and it is
worthless.

Across eighteen reviewer dispatches in this session, nine through the Agent tool, nine headless, I asked
every single one to self-report its model. **Exactly one of the eighteen did it.** One. The agents with strict
output formats, the reviewers that return structured findings, which is to say the useful ones, swallowed
the self-ID line entirely; it did not fit their schema, so it never appeared. The lone reviewer that did emit
its ID happened to be accurate, which is the cruelest part: a 1-in-18 compliance rate that is *correct when it
fires* is more dangerous than one that is wrong, because it tempts you to trust the channel.

Self-identification cannot be the provenance mechanism, and the reason is structural, not fixable with better
prompting. You're asking the component under test to certify its own identity, through the same fallible
output path whose reliability is the thing in question. The agent that is misconfigured is exactly the agent
whose self-report you cannot trust, because the misconfiguration is upstream of the report. It's a witness
vouching for itself.

The provenance has to come from *outside* the agent's output, from the harness's own accounting of what it
ran. In Claude Code that channel is `modelUsage`: the result event of a headless run records, as ground truth
the agent never touches, which model actually consumed the tokens. For Agent-tool dispatches the equivalent is
the transcript file on disk, `agent-<id>.jsonl`, with `.message.model` on every turn. Those are the
instruments that don't lie, because the agent isn't the one writing them. Every number that survived this
session was re-grounded against `modelUsage` or the transcript. None of them rest on what an agent said it was.
This is Chapter 3's discipline turned from the guard onto the gauge: you do not ask the thing to certify
itself. You read the record that was written about it by something it cannot edit.

## The clean re-run, and the ceiling nobody wanted

The operator removed the pin mid-session. (`~/.claude` is a protected target, so the change was staged to a
review file, `staging/remove-subagent-model-pin.md`, and applied by hand, with the prior settings backed up
to `settings.json.bak-pin` first. I have since confirmed the variable is gone from the live `env` block as of
2026-06-12; the pin is not still running.) With it gone, I re-ran the matrix the honest way, through a
headless driver, `run_headless_baseline.py`, that never depended on the Agent-tool path and dispatches each
model directly, with `modelUsage` verifying every single run. Nine cases, twenty-one planted bugs, every cell
provenance-checked. Here's what the uncontaminated data looks like:

| Model | Recall | False positives |
|---|---|---|
| Fable | 21 / 21 | 0 |
| Opus 4.8 | 20 / 21 | 0 |
| Haiku 4.5 | 20 / 21 | 0 |
| Sonnet 4.6 | 19 / 21 | 0 |

Read that table, because the headline is not in the order. The headline is the *spread*.
From Haiku to Fable, the entire range is two planted bugs out of twenty-one, with zero false positives at
every tier. **The review ceiling is saturated.** On this task, finding planted defects in a diff and not
crying wolf, the cheapest model in the lineup and the most expensive one are separated by a margin so thin it
disappears under inspection.

Because I did inspect it. Every recall difference in that table is a *reporting-style artifact*, not a
detection difference. The most telling example: one planted bug was a `curl -fsSL … | bash` install pattern,
and the scorer was keyed to the exact string `curl -fsSL`. The Fable-lane run described the attack as
`curl | bash`. Haiku and Sonnet described the structural vulnerability correctly but never typed the literal
flags the scorer was hunting for. Every model *saw* the bug. The scorer, matching on a brittle substring,
recorded a miss for the ones that paraphrased. The two-bug "gap" between Fable and Sonnet is not a gap in
security reasoning. It is a gap between which models happened to phrase their findings in the vocabulary my
scorer was grepping for, which is a fact about my scorer, not about the models.

So the eval, having just caught itself lying through a config pin, immediately revealed a *second* way it had
been misleading me: a scorer that rewards the spelling of a finding over the substance of it. That's two
distinct apparatus failures surfaced in one session, and neither was about the agents. Both were about the
thing measuring the agents.

## What it cost to actually know

There's a number I want to put next to all of this, because it reframes the whole exercise: the clean,
provenance-verified headless re-baseline, the nine runs that produced the honest table above, **cost about
$2.10.**

Two dollars and ten cents. That's the entire price of replacing two months of confident, fabricated,
plausible data with two dollars of data I can actually defend. I want that number to sting a little, in both
directions. It should sting because of how cheap the truth was relative to how long I went without it, the
contamination didn't persist because verification was expensive, it persisted because I never spent the two
dollars to check. And it should sting because of how cheap it makes the alternative look in hindsight: I had
been making routing decisions, forming opinions about model quality, carrying "Sonnet is as good as Opus at
review" around in my head as a learned fact, and the cost of grounding that belief was less than a cup of
coffee the whole time.

The lesson isn't "evals are expensive so be careful." It's the opposite. Evals are *cheap*, and provenance
verification is cheaper, and the only thing standing between you and the truth was the discipline to not trust
a labeled number until you'd checked who actually produced it.

## Why agent evals are not model evals

This is the chapter's real argument, and the contamination is just the most vivid way I know to make it.

When a lab evaluates a model, the model is the system under test and the harness is assumed correct. That
assumption is mostly fine for them, because they built the harness to be the stable, trusted part. But when
*you* evaluate an *agent*, a model wrapped in your instructions, your tools, your routing config, your
scorer, your fixtures, the thing under test isn't the model. It's the whole apparatus, *including the parts
you built to do the measuring.* The model is one component in a stack of components you assembled, and every
other component in that stack is an artifact that can drift, go stale, or lie, exactly like the guard in
Chapter 3 and the dossier in Chapter 4.

The config pin was part of the system under test and I had mistaken it for part of the trusted substrate. The
brittle scorer was part of the system under test and I had mistaken it for the neutral judge. The
self-identification line was part of the system under test and I had mistaken it for ground truth. Three times
in one session, the thing I was relying on to tell me the truth about the agents was itself an unaudited
agent-environment artifact, and three times it was the thing that was actually wrong.

So the discipline that comes out of this chapter is narrow and absolute: **a model-comparison number is
worthless until its provenance is verified from a channel the agent cannot edit, and a scorer's verdict is a
lead until you've read the transcript to see whether a "miss" was a real miss or a vocabulary mismatch.** The
measurement is not the end of the inquiry. The measurement is the next thing to be measured. You audit the
gauge with the same suspicion you brought to the agent, because the gauge is just another artifact you built,
and you already know, from every other chapter in this book, how the artifacts you build behave when you
stop watching them.

The first time the agent lied to me, in Chapter 1, it did so by changing a test to assert broken behavior and
truthfully reporting that the assertion passed. The measurement in this chapter lied the same way, one level
up: it truthfully reported the score of a model that was not the model I thought I was scoring. Both were
accurate. Both were wrong. And the only thing that ever tells accurate from correct is the verification you
run on your own instruments, after you've stopped trusting them, with two dollars and the willingness to look.