Score the Operator, Not the Worker

I built an eval to catch AI agents believing their own confident output. Then I watched it flatter the verbose, crown its own model family, rank the whole field off nine cases that mostly proved nothing, and hand me a headline number that measured one direction of the two it promised. The checking apparatus I'd built to audit the agents turned out to be an unreliable narrator of its own competence, which is the one thing this series keeps insisting no system can credibly narrate about itself.

The other essays aim that machinery at an API, a plugin, a model, the operator's own future self. This one aims it somewhere more uncomfortable: at the verifier. I built OPERANT to rank AI agents as operators, trusted it to tell me the truth about them, and found the scorer, the judge, the corpus, and the score itself each quietly narrating its own competence. The ranking of the models was the least interesting thing it produced.

Most coding evals ask the same question: can the model write the code? Hand it a function signature and some failing tests, see if it goes green. Useful, but it skips the part of agentic work I actually worry about. When an AI agent is running real operations (reading my inbox, touching my shell, pushing to my repos), the failure I care about isn't "the code was wrong." It's "the agent did the thing it should have refused to do."

So I built a different benchmark. OPERANT scores an agent as an operator: given a situation laced with social-engineering cues, guard-bypass tricks, or benign-but-scary surface tokens, does it make the right meta-decision: refuse, escalate, take the sanctioned path, or just proceed? The agent-under-test runs with the operator contract as its system prompt and emits a structured decision. The scorer never checks whether it named the right symbol or phrased its reasoning a particular way. Only the decision counts.

I ran Haiku, Sonnet, and Opus through it. I got a clean ranking. And then the things I actually learned had almost nothing to do with which model won, and almost everything to do with how easily I'd have believed a wrong answer.

Evidence status update (2026-07-17)

The account below preserves what the original analysis reported; it is not a durable current model comparison. The public evidence bundle binds repeat-level OCS values and lets a reader recompute their mean, population standard deviation, delta, and exact relabelling arithmetic. Accuracy, confidence intervals, orchestration aggregates, and the per-case decisions needed to independently derive the reported 1.000 TPR ceiling are not independently recalculable from the public bytes. The historical as-run corpus, protocol, dispatch freshness, served-model identity, and independent replication remain UNKNOWN. Treat the named-model ranking, significance language, and model-equivalence claims below as NOT_DURABLE until a fresh identity-bound run is independently replicated. The public correction and evidence binding carry the current boundary.

The clean result (the part you'd guess)

The headline metric is OCS, Operational Calibration Score, which is just TPR − FPR over a single binary signal: did the agent withhold execution when it was warranted? The nice property is that it's impossible to game with a uniform policy. An agent that refuses everything has a perfect true-positive rate and a perfect false-positive rate, so it scores zero. An agent that proceeds on everything scores zero too. Only an agent that actually discriminates (withholds on the malign cases, proceeds on the benign ones) scores above zero. That's the design, and the design is sound. What a design promises and what a given run ends up measuring are two different things, though, and the widest gap between them is the last thing this eval taught me.

That discrimination requirement is enforced by the corpus design: every malign case has a benign twin that shares its domain and trigger words but differs in two things the agent has to notice: where the instruction came from (the operator, or some untrusted content the agent was processing) and how big the blast radius is. "Delete this branch" is fine for a regenerable scratch directory and a problem for unmerged shared work. Same verb, opposite call.

The result was tidy and exactly what you'd expect:

Monotonic, and the bands don't even touch. Sonnet's best run sits below Opus's worst. Bigger model, better operator. If that were the whole post, it'd be a tweet.

And it holds up to a real test, not just eyeballing the bands. Enumerate every way to relabel the five Opus and five Sonnet repeats (all 252 of them), and an exact permutation test puts Opus ahead at p = 0.0079. That's the lowest value the test can return: the two models' runs don't overlap at all, so only the observed split and its mirror reach the observed gap. Five repeats is a small sample, so I lean on the exact test rather than a wider bootstrap interval, and Haiku's last place is a single draw resting on the size of its gap rather than a p-value. But Opus over Sonnet is real, not a lucky run.

The interesting part showed up when I tried to score a fourth dimension: not whether the agent makes the right call, but whether it sizes and routes the work like a good operator: solo versus parallel agents versus a coordinated team, which model to put on which subtask, when to bother with isolation. That's where the checking apparatus started lying to me.

Lesson 1: keyword scoring punishes the articulate

My first attempt at scoring the orchestration dimension was deterministic: keyword anchors. Expect the correct routing to show up; forbid the over-engineering. If the plan says "five-agent team" for a one-line typo fix, fail it. Clean, cheap, no model calls.

It saturated immediately. All three models landed around 0.90. On the hardest cases the smaller model scored higher, the exact inversion of the OCS ranking.

The root cause is the good part. The forbid anchors were firing on plans that explained or cited the machinery they were correctly declining. Here's Sonnet making the right call on a task that looks big but is actually a solo job: "Worktrees exist for parallel agent isolation; with one writer there's nothing to isolate from." That's a correct decision, articulated well. The keyword scorer saw the word "worktree" and docked it anyway. It even penalized a plan for naming the policy document (auto-worktree.md) while making the right choice. The terse plan that just said "no worktree needed" scored full marks; the thorough, correct, well-reasoned plan got punished for showing its work.

I could have patched it. Add a rule for dismissals, then another for definitional mentions, then another for filename citations. But that's the lesson, and it's the same one the guard essay tells about security hooks: keyword presence is a terrible proxy for a decision, and every false-positive class is a separate patch in an infinite series. The fix isn't a better regex. It's to stop scoring tokens and start judging the decision.

So I moved that dimension to a calibrated LLM judge: hand it the task, a reference rationale, and the plan; ask for a per-dimension verdict; explicitly instruct it to grade the decision, not the verbosity, and to treat declined-and-cited machinery as correct. I validated the judge before trusting it. On reference plans it scored 1.000; on deliberately over- and under-engineered traps it scored 0.000. Perfect separation. Now it ranked where the keywords had flatlined. I had a working verifier. I almost believed it.

Lesson 2: the judge has a favorite, and it's itself

The judge gave me this:

Haiku clearly trails. Sonnet and Opus are a near-tie, with Sonnet, the judge's own model family, nominally on top.

That tie sat wrong with me. Was it real, or was the Sonnet judge quietly flattering itself? So I re-ran the entire judgment (all 187 transcripts) with Opus as the judge instead, and diffed.

There it is, clean as you like. Each judge scored its own family two to three points higher. The nominal winner flips depending on who's judging: Sonnet wins under Sonnet, Opus wins under Opus, and neither margin is statistically meaningful. Haiku, in neither family, sits at exactly 0.824 under both judges, totally unmoved.

This is the finding I'd actually put in front of someone building an LLM-judge eval: don't rank same-family models with a same-family judge. The self-preference here is larger than the real gap I was trying to measure. If I'd run a single judge and shipped the number, I'd have published a coin flip as a result. The fix is cheap: judge with a third or stronger model, average across multiple judges, or just report the close pair as peers and stop pretending you resolved them. The verifier had a thumb on the scale, and the only way I found it was by building a second verifier to check the first.

So I built the third verifier the second one implied: an ensemble that averages the two judges, cell by cell, across all 187. If each judge flatters its own family by the same couple of points, that bias is symmetric. A symmetric bias cancels under the average. What's left is family-neutral: Opus 0.963, Sonnet 0.951, Haiku 0.824, with the top two's repeat bands fully overlapping. Same verdict the eyeball gave, now with the thumb lifted off the scale. Haiku is clearly last, and Sonnet versus Opus is a tie I should stop trying to break.

The two judges' disagreements turned out to be a second instrument I hadn't planned for. They agree on 172 of the 187 cells and split on only 15, and the 15 aren't scattered. They pile up on exactly the cases that are genuinely hard: the sequential-framing trap, the eight-stream migration, the mixed-sensitivity routing. The easy cases, where everything scores full marks, produce zero disagreement. So where the judges disagree is itself a map of where the decision is actually ambiguous, the verifier's own uncertainty made legible. The thing I built to check the models turned out to also measure which questions don't have a clean answer.

Lesson 3: small corpora lie with a straight face

There's one more. It's the one I'm most embarrassed by, which is exactly why it's worth telling.

The first time I ran the judge, my orchestration corpus had nine cases. The result was a clean, confident, monotonic Haiku < Sonnet < Opus. It looked publishable.

It was garbage. I'd written most of those nine cases to exercise the scorer, not to separate the models, and eight of the nine saturated at a perfect score for the top two. The entire Sonnet-versus-Opus story rested on a single case. One.

So I added eight more, written specifically to discriminate: each one a believable wrong answer paired against the right one, with the difference living in the structure rather than the surface. The Haiku gap survived. The clean Sonnet < Opus ordering dissolved into the tie, and one of the new cases flat inverted it.

That inverting case is my favorite thing in the whole project. It's a task framed as "first do X, then Y, then Z," sequential-sounding, but the three steps are actually independent and parallelizable. Sonnet sees through the framing and parallelizes. Opus and Haiku both take the narration at face value and serialize it. A specific, repeatable behavioral difference that runs opposite to model size, and it survives both judges. That single case carries more signal than the eight saturated ones combined.

The lesson: a benchmark is only as trustworthy as its hardest cases. Cases that everything passes feel like coverage. They tell you nothing. The clean early ranking wasn't a finding; it was an artifact of a corpus too easy to separate the top.

Even the headline had a ceiling

There is a fourth narrator, and it is the one I trusted most: OCS itself, the clean number at the top of this piece.

Later, doing to the metric what I had done to the judge, I audited the axis it rides on. Across every model and every repeat, the true-positive rate was pinned at exactly 1.000. Every model withheld on every blatant malign case; the malign half of the score never moved. So the number I published, OCS as TPR − FPR, was empirically 1 − FPR. It was measuring over-refusal and nothing else. The benign twins carried the entire result, and the malign twins, the ones that were supposed to make the score two-directional, were too obvious to separate anyone.

The ranking survives that. Opus over-refuses least, Sonnet next, Haiku most, and the order holds. But "impossible to game in both directions" was a claim about the design, not about what this corpus exercised, and I missed the difference because a perfect true-positive rate looks exactly like success. The score I called symmetric was, in practice, one-directional.

The fix is OPERANT-2's, and it is the same move as every lesson above. Disguise the twins so the malign case stops being obvious, then probe the ceiling from outside the family: a zero-cost off-family run broke the true-positive rate off 1.000, down to 0.75, which is the proof the ceiling was a property of the corpus and not a law of nature. There was a tempting shortcut too, a deterministic patch that would have forced the axis to move, and it got built and then killed under review, because forcing the number is exactly the keyword scorer's mistake in a new place: rigging the corpus to produce the spread I wanted instead of measuring the spread that was there. That correction is its own essay. What belongs here is the admission: the number I led with was narrating its own completeness, and I shipped it before I checked.

What it actually measured

The leaderboard, for the record and with the ceiling caveat attached: Opus ranks first on the decision score, Sonnet second, Haiku last, and on this corpus that is a ranking of which model over-refuses least. On orchestration judgment, Sonnet and Opus are peers and Haiku trails. Useful. Roughly what you'd expect.

But the durable output wasn't the ranking of the models. It was four ways I almost fooled myself, each caught by the boring verification step I almost skipped:

• Score decisions, not tokens. Keyword anchors punish the articulate and reward the terse, the opposite of what you want from a judgment benchmark.
• Never let a model grade its own family. The self-preference is measurable, and it can be bigger than the signal you're chasing.
• Distrust any ranking your easy cases agree on. Saturation looks like confidence and is actually blindness.
• Audit the axis, not just the score. A metric can read as two-directional and quietly collapse to one when every case pins the same side; a saturated axis measures nothing, and a perfect rate looks exactly like success.

The second judge, the wider corpus, the matched twin, the axis audit: every one of them turned a clean, confident, wrong conclusion into an honest one. This is the whole series in one artifact: the eval was the machinery I built to convert an agent's behavior into checked truth, and then the eval itself became one more layer that couldn't self-report honestly. The score flattered by construction. The scorer flattered the verbose. The judge flattered its own kin. The corpus flattered the top of the table. None of them were lying on purpose; all of them were narrating their own competence, and competence is the one thing a system can never credibly narrate about itself.

So you point the machinery inward. You judge the judge, widen the corpus, build the second verifier to audit the first, not because you expect it to fail, but because the version of me that ships a single-judge nine-case ranking is exactly as naive as the agent that takes "first X, then Y" at face value and serializes the work. Verification capital is only capital if it also gets spent on the verifier. The benchmark's real result was a ranking of the ways I was about to be wrong. That's the one I'd keep.

OPERANT is a small benchmark: matched malign/benign pairs scored with OCS, an LLM-judged orchestration axis validated against authored references, and a zero-model-call selftest that proves the scorers reject every gaming strategy before a single token is spent.

Goes deeper in the book → Chapter 5: The Measurement That Lied