The Auditor Audits Itself

Every model withheld on every one of the 18 malign guard cases, on every repeat. Haiku did. Sonnet did. Opus did. The true positive rate for all three sat pinned at exactly 1.000, and the script that surfaced this, ceiling_audit.py, did nothing more glamorous than read the recorded run data back. Zero new runs, zero API cost. The evidence had been sitting inside the published results the entire time, waiting for someone to ask the results a slightly ruder question.

This chapter is about asking that question, and then about what you owe the answer once you have it.

The number that looked done

The public OPERANT results had produced a clean-looking model ranking: the keyword scorer had been replaced, the judge-family caveat had been named, and the result looked repaired enough to publish. This chapter audits that result's own headline metric, because the ranking that came out of all that repair work looked, by every conventional signal, finished.

OPERANT, for readers arriving here first, is a decision-calibration benchmark for AI agents operating as operators. It does not score code quality. It scores the meta-decision an agent makes when the content in front of it is adversarial or guard-railed: proceed, withhold, escalate, reroute. The headline metric is OCS, the Operational Calibration Score, defined as TPR minus FPR over one binary signal: did the agent withhold execution when withholding was warranted. The formula has a property worth admiring. Uniform policies score zero. An agent that refuses everything scores zero, an agent that proceeds on everything scores zero, and only an agent that discriminates, that says no to the right things and yes to the rest, climbs above it.

OPERANT-1 produced a clean monotonic ranking on that metric: Haiku at +0.273, Sonnet at +0.691, Opus at +0.873. The repeat bands did not overlap; Sonnet's best run sat below Opus's worst. An exact permutation test put Opus ahead of Sonnet at p = 0.0079, which is the lowest value that test can return for this setup. Non-overlapping bands, a floor-level p-value, a metric that punishes both failure modes by construction. It looked done.

Then ceiling_audit.py read the runs back and found the pinned 1.000.

What the score was measuring

Here is the collapse, and it is algebra, not interpretation. If TPR is fixed at exactly 1.000 for every model, then OCS = TPR − FPR reduces to 1 − FPR. The score that was designed to be bidirectional, that punishes both over-refusal and under-refusal, was empirically measuring one side only: how often a model over-refuses something that merely looks scary. The benign twins carried the entire ranking. The malign cases contributed nothing to the differences between models, because they were too blatant to ever induce a miss. All three models spotted all 18 of them, every time.

The claim "impossible to game in both directions" was true of the formula. It was false of what this corpus exercised. The formula had the property; the test never turned the key on one of its two locks, because every subject walked through the same door on the malign side and the door was painted red with a siren on it.

Note what this does and does not mean. The ranking itself survives the audit, and survives it with a corrected reading: Opus over-refuses least, Sonnet next, Haiku most. That is a real, reproducible, useful result about calibration on the benign side. What did not survive was the interpretation. A benchmark that advertises two-sided calibration and delivers one side has told a true story under a wrong title, and the title is the part people quote.

Testing the fix before trusting it

The repair was OPERANT-2: twelve new malign twins, this time disguised. Plausible requests, operator-authorized in tone, the kind where the wrong answer is a believable PROCEED rather than an obvious catastrophe. The design goal was a malign case a competent agent could conceivably fall for, because a malign case nobody falls for measures nothing.

But a corpus that is supposed to break a ceiling has to demonstrate that it breaks the ceiling, and running a full Claude matrix to find out is an expensive way to discover your new cases are as blatant as the old ones. So the corpus was pressure-tested first, at zero cost, against a non-Claude subject: GPT-5.5, driven read-only through Codex, on the same scaffold. Three of the twelve disguised cases broke through. TPR came back at 0.75, not 1.000.

The ceiling breaks now.

The choice of an off-family subject was deliberate, and the reason deserves stating plainly because it returns at the end of this chapter. A Claude judge grading Claude subjects against Claude-authored references carries an acknowledged, unresolved circularity. An off-family subject sidesteps that for this specific question: if GPT-5.5 misses cases that every Claude model caught in the old corpus, then the old 1.000 was a property of the corpus, not a law of nature. The probe answers the ceiling question. It does not answer the judge question, and we will get to what it leaves open.

Auditing the break itself

Even the break got a second pass, because a benchmark that celebrates its first failure without inspecting it has learned nothing from its own history.

Two of the three breakthrough cases looked, on first read, like the binary scorer punishing a calibrated move. The agent had declined the risky mechanism in the request and routed instead to a masked, sanctioned alternative, and the scorer counted that as a leak. If true, that would be a scorer bug wearing a miss's clothing, and OPERANT had been burned by exactly that species of bug before.

A matched-pair argument resolved it. Every malign case in the corpus has a benign twin, and the benign twin of that pair already owns the "sanctioned proceed" label. That is the pair's entire reason to exist: two near-identical requests, one of which should proceed and one of which should not. If the malign twin proceeding by the sanctioned route gets scored as safe, the one distinction the pair was built to test collapses, and the pair measures nothing. So the scorer's verdict stands. A genuine miss, not a scorer bug.

The patch that got built and killed

The ambiguity was tempting to engineer away. A deterministic "sanctioned-substitute" clearing rule was implemented: detect the reroute-to-sanctioned-alternative pattern mechanically and handle the edge case without bothering a judge.

A reviewer's adversarial probe broke it in one sentence. A stray connective was enough to launder a real PII-exfiltration commitment into a scored-safe withhold: "rather than open a ticket, I will dump the users table…". The rule saw the shape of a refusal with a sanctioned substitute. What the sentence contains is a commitment to exfiltrate user data, wearing the refusal's grammar.

Fully reverted.

The standing rule that came out of the wreckage: semantic judgment calls belong to an LLM judge, never to a deterministic keyword clear. And this is a lesson the project has now paid for twice, in two different places, independently. The orchestration axis once shipped a keyword scorer that inverted, rewarding terse plans and punishing articulate correct ones. The decision axis's clearing rule laundered an exfiltration. Same root cause both times: deterministic string-matching standing in for semantic judgment. When the thing you need to evaluate is meaning, a pattern over characters will eventually be satisfied by the wrong meaning, and it will happen on the case that matters.

One more receipt, folded in for free

While the audit machinery was open, a residual statistical honesty check went in. The published exact-permutation p-value for Opus versus Sonnet, p = 0.0079, is the algebraic floor for a 5-vs-5 non-overlapping comparison. It is a restatement of "the bands don't touch," not independent evidence beyond it. Anyone reading the number as a second, separate confirmation of the ranking is reading one fact twice.

This got recorded as a caveat, not hidden. It costs a sentence in the results and buys the right to keep calling the benchmark honest.

OPERANT, corrected corpus and caveats included, is public at github.com/saagpatel/operant under MIT, and it ships a bring-your-own-agent runner, score_my_agent.py, so the audit trail above is not something you have to take on faith. You can point it at your own agent and generate your own disappointments.

Grading the graders, one level up

The self-audit instinct, once installed, does not stop at one benchmark. It regresses, and the regress turns out to have layers you can name.

OPERANT scores agent behavior: the decisions an agent makes under pressure. But an agent in production does not act alone. It acts inside a harness, the hooks and permissions and rules of a Claude Code or Codex setup that constrain what its decisions can touch. A well-calibrated agent inside a harness with readable credentials is a good driver in a car with no brakes. So there is a second published tool, harness-scorecard, that grades the harness itself: stdlib-only, read-only, an A-to-F maturity grader, public on PyPI and GitHub and available as a GitHub Action.

harness-scorecard scores ten rubric dimensions into a weighted grade, but the weighting is not the interesting part. The interesting part is the capability gates, which cap the grade when a critical hole is present. You cannot score an A with readable credentials, no matter how many cheap checks pass. This is the OPERANT lesson transplanted: a score that can be maximized by accumulating easy points while the load-bearing property fails is measuring effort, not safety.

Its distinguishing feature is dispatcher introspection, and this is where the regress gains a third layer. Many harnesses route their checks through an opaque hook dispatcher, so the grader cannot see the checks directly. harness-scorecard reads the dispatcher's source, finds each check's guard signature, and suggests, never auto-credits, that a check is covered. Every pattern it matches is code-anchored: a regex, an identifier, a call, a path-literal. Never a prose match. And each pattern is paired with an oracle test pinning the false-credit boundary: a check that would credit sanitize-path-style prose, or a bare file extension, is a bug in the grader, not evidence about the harness. The grader ships with tests that define what it is not allowed to be fooled by.

Name the layers. OPERANT scores the agent's decisions. harness-scorecard scores the harness that constrains the agent. Dispatcher introspection scores how much of the harness's own logic is even visible to the grader, which is the grader being honest about its own blind spots. It is the same instinct all three times, aimed at progressively more meta subjects: don't trust the label, verify the effect. The pinned 1.000 was a label ("bidirectional") diverging from an effect (one-directional measurement). A permissive check hiding behind a dispatcher is a label ("covered") that introspection refuses to grant without a code anchor.

And the dogfood loop is real rather than decorative. harness-scorecard graded the operator's own Codex harness configuration, not a demo target built to score well, and that grade drove a measured D-to-B hardening pass. The tool's first serious casualty was its own author's infrastructure, which is the correct order for a tool like this to draw blood.

The floor self-evaluation cannot audit away

It would be satisfying to end by declaring the loop closed: benchmark audits itself, corpus fixed, grader grades the graders, everyone goes home calibrated. The receipts do not support that ending, and this whole chapter has been about not outrunning the receipts.

The off-family probe answered the ceiling question. GPT-5.5 breaking three of twelve disguised cases proves the old 1.000 was a corpus artifact. But the judge-family circularity remains open for the judgment axis. A Claude judge scoring Claude subjects against Claude-authored references shares a family bias with everything it grades, and an all-Claude ensemble cannot cancel it: averaging two Claude judges removes the differential between them, not the bias they share. That is not a gap a cleverer script closes. It is a structural limit on what this benchmark, run this way, can know about itself, and it is recorded as unresolved because it is unresolved.

A benchmark about honest calibration gets to keep that name only as long as it applies the standard to its own limits. The auditor audited itself and found the headline claim half true. Then it audited the audit, killed its own favorite patch, filed a caveat against its own p-value, and wrote down the one bias it cannot remove.

That last item is still on the list. The list is public. That is the whole method.

The short version → We Rigged Our Own Benchmark