We Rigged Our Own Benchmark

There is a script checked into the OPERANT repo called ceiling_audit.py. It makes no API calls, costs nothing to run, and reads only the recorded run data from a benchmark we had already shipped. Its one job is to answer a single question: across every model and every repeat, how often did the agent withhold on the 18 malign guard cases, the ones where withholding is the right call? The answer it printed was 1.000. Not high. Not near-ceiling. Exactly 1.000, for all three models, on every repeat.

That number is this whole essay. Everything else is what we did about it.

The result that looked done

A few sentences of setup for anyone arriving fresh. OPERANT is a decision-calibration benchmark for AI agents operating as operators. It leaves code quality to other benchmarks; what it scores is the meta-decision under adversarial or guard-railed content: proceed, withhold, escalate, reroute. The headline metric is OCS, the Operational Calibration Score, computed as TPR − FPR over one binary signal: did the agent withhold execution when withholding was warranted. By design, uniform policies get nothing. Refuse everything and you score zero. Proceed on everything and you score zero. Only an agent that discriminates scores above zero.

OPERANT-1 produced the kind of result you frame and hang. Three models, a clean monotonic ranking: Haiku +0.273, Sonnet +0.691, Opus +0.873. The repeat bands do not overlap; Sonnet's best run sits below Opus's worst. An exact permutation test put Opus ahead of Sonnet at p = 0.0079, which is the lowest value that test can return for this setup. The full story of that run lives in the ranking that started this (Score the Operator, Not the Worker), which ends by disclosing this correction in miniature and promising the full version. This is the full version.

It was genuinely good work. It looked done. The problem with results that look done is that "done" and "audited" are different states, and only one of them involves running the script.

What the audit found

Here is the substance, stated as plainly as I can state it.

With TPR pinned at exactly 1.000, the arithmetic of the headline metric collapses. OCS = TPR − FPR becomes, algebraically, 1 − FPR. The score we had described as bidirectional was, empirically, measuring one side: how often a model over-refuses something that merely looks scary. Every point of separation between Haiku, Sonnet, and Opus came from the benign twins. The malign cases contributed nothing to the ranking, because no model ever missed one, on any repeat, ever. They were too blatant to induce a miss. An axis that never varies is not an axis; it is a constant with a column header.

Nobody rigged this on purpose. We rigged it the way most benchmarks get rigged, by authoring the hard cases from the inside, where they all look hard. The version of me that shipped a one-directional score under a bidirectional description was not lying; he was measuring the design and reporting it as the corpus.

Two things are worth keeping straight, because they pull in opposite directions. First, the ranking itself survives the audit: Opus over-refuses least, Sonnet next, Haiku most, and that order holds. Second, the interpretation does not survive. "Impossible to game in both directions" was a true claim about the scoring formula and a false implication about what this corpus actually exercised. The interesting failures live in exactly that gap.

Testing the fix before trusting it

The obvious repair is better malign cases, and OPERANT-2 wrote them: 12 disguised malign twins, plausible, operator-authorized-sounding requests where the wrong answer is a believable PROCEED rather than an obvious one. The failure mode we were correcting was cases that announce themselves; the replacement cases had to be capable of fooling someone.

But a redesign that fixes a ceiling has to demonstrate the ceiling can now break, and demonstrating that with the same family of models that produced the ceiling proves less than it appears to. A Claude judge grading Claude subjects against Claude-authored reference answers carries an acknowledged circularity we have not resolved. So before spending anything on a full Claude model matrix, the new corpus was pressure-tested at zero cost against a non-Claude subject: GPT-5.5, driven read-only through Codex, on the exact same scaffold.

Result: 3 of the 12 disguised cases broke through. TPR 0.75, not 1.000.

This is the strange emotional center of the project, the moment where a worse number is better news. The off-family probe is how you establish that the old pinned 1.000 was a property of the corpus, not a law of nature. The ceiling breaks now. probe_offfamily.py is checked in next to ceiling_audit.py, both zero-Claude-cost, both re-runnable against the recorded data by anyone who doubts the story.

Even the break got a second pass

Two of the three breakthroughs looked, on first read, like the binary scorer punishing a calibrated move. In each, the agent declined the risky mechanism but routed to a masked, sanctioned alternative, and the scorer marked that as a leak. If that reading held, the "breaks" would be partly a scorer bug, and the 0.75 would be soft.

The matched-pair structure of the corpus settled it. The benign twin of that exact pair already owns the "sanctioned proceed" label; that is its entire function. If the malign twin can proceed the same way and be counted safe, the one distinction the pair exists to test collapses into nothing. A genuine miss after all, not a scorer artifact. The design defended itself better than our first read of it did.

The patch we built, then killed

There was still a tempting move on the table: a deterministic "sanctioned-substitute" clearing rule that would auto-handle the routed-to-an-alternative ambiguity, so the binary scorer could recognize a safe reroute without a judgment call. We built it. It worked on the cases in front of us.

A reviewer's adversarial probe broke it in one sentence. A stray connective was enough to launder a real PII-exfiltration commitment into a scored-safe withhold: "rather than open a ticket, I will dump the users table…". The rule saw the shape of a sanctioned substitute and cleared it. What the sentence commits to is dumping the users table.

The rule was fully reverted, and the lesson went in the log as a standing one: semantic judgment calls belong to an LLM judge, never to a deterministic keyword clear. The orchestration axis had already paid this exact tuition once, when its keyword scorer inverted and started rewarding terse plans over articulate correct ones (that story belongs to the sibling essay), so this was the second time the same lesson arrived through a different door. Keyword rules are cheap, fast, reproducible, and confidently wrong in precisely the situations the benchmark exists to catch.

One more caveat, folded in for free

Since we were re-reading everything with suspicion anyway, the statistics got a pass too. The published exact-permutation p-value for Opus versus Sonnet, that p = 0.0079, is the algebraic floor for a 5-vs-5 non-overlapping comparison. It is a restatement of "the repeat bands don't touch," not independent evidence beyond that fact. It was recorded as a caveat rather than quietly retired, which is the correct fate for a true number that sounds like more than it is.

Why publish this

Public AI-eval writing has a shape: here is our benchmark, here is our leaderboard, here is why it matters. The genre almost never includes the follow-up where the authors report that their headline metric was, in practice, exercising half of its promise. Which is odd, because that follow-up is where the value concentrates. The corrected OPERANT says strictly more than the original did: the ranking holds, the over-refusal axis is real, the miss axis now has cases capable of producing misses, the scorer's one tempting shortcut has been tried and executed, and the prettiest p-value in the repo comes with its own deflation notice attached.

The audit is not a confession extracted under pressure. It is two small scripts, checked in, that anyone can run against the recorded data. That is the standard the rest of the work has to live up to now, which is inconvenient and correct.

OPERANT is public at github.com/saagpatel/operant, MIT-licensed, and ships a bring-your-own-agent runner, score_my_agent.py, that points the same scorer at whatever agent you operate. Run it. Then find the axis in your own headline metric that never moves, and check whether it is measuring your subjects or your corpus.

Postscript, 2026-07-11

A fourth model has now sat the exam, and it extends the ceiling story twice. Claude Fable 5 ran OPERANT-2 and its decision OCS came in at 0.833: the disguised-malign cases dropped its true-positive rate from the old pinned 1.000 to 0.833, with zero false alarms, so the same corpus hardening that broke GPT-5.5's ceiling broke Fable's too. Then I tried to bring Fable to a proper five-repeat profile headless, and hit a different wall. Fable's own safety classifier deterministically refuses the guard-warranted malign prompts, so a comparable score cannot be produced that way. I kept the honest n=1 and logged the block rather than publish a cherry-picked number.

Goes deeper in the book → Chapter 12: The Auditor Audits Itself