The Benchmark That Argued Back
I built a benchmark that puts models in the judge's seat: verdicts, calibrated confidence, principled refusal, survival under challenge. Its first sitting scored two models, and three times the apparatus turned on its own author: the scorer lied unanimously, a test case got overturned by the defendants, and both models flagged a standing ruling in my own records. All three reversals were the system working.
Hours ago I published a confession about the courts I build: calibrated judgment instruments that rule on my own work, and the audit that caught four of them forging their own graduation certificates. That essay ended with a rule. Build the court, then audit it like the defendant owns it, because one of you does.
Last night I built the next instrument under that rule, and its first sitting is the fastest I have ever gone from “the apparatus works” to “the apparatus is arguing with me.” It argued back three times. All three times it was right, and that is the story, because a measurement system that can win an argument against its own author is the only kind worth publishing numbers from.
The judge’s seat
The benchmark is called OPERANT-J. Its predecessors measure a model as an actor: can it do the work, and does it refuse malign work. OPERANT-J measures the model as a judge. Each of 18 cases hands the subject a claim and a bundle of artifacts: real adjudicated case files from my operating courts, plus authored controls. The subject must return one of three verdicts. Upheld: the artifacts support the claim. Defective: they contradict it. Or a third verdict that most benchmarks never score: this bundle cannot carry a ruling either way, and here is the specific gap.
Four cases are built so that refusal is the right answer: a report resting on an attached log that is not attached, a citation to lines 88 through 104 of a file, an improvement claim standing on one run before and one run after. Ruling on those is an error. Refusing to rule on the fourteen rulable ones is also an error. Timidity and bravado price each other.
Every verdict carries a confidence, scored with a proper scoring rule, so hedging everything and swaggering everything both lose measurably. Every citation is checked against the actual bundle bytes. And after each verdict, the judge gets one scripted challenge arguing the opposite reading, scored on a matrix where capitulating on a correct ruling and standing on a wrong one both cost you. Conviction is only worth points when it is grounded.
Two models sat. The expensive one went 16 of 18, with a Brier score of 0.10, which is to say its confidence meant something: its two misses arrived at its two lowest stated confidences. The cheaper one went 13 of 18 at 0.17, and twice retreated to “cannot rule” on cases that could be ruled. Both went a perfect six for six on my planted traps, the seeded defects and the clean-but-suspicious-looking controls. The entire gap between them lived in the eight real court cases, the messy adjudicated material with operator fingerprints on it.
That is finding one, and I want to underline it because it surprised me: the synthetic cases, the ones I crafted to be discriminating, discriminated nothing. Real cases carried all the signal. If you are building an eval, the artisanal traps are your floor check. The ceiling lives in your production record.
Three arguments, three losses, all mine
The scorer lied unanimously, and an old rule caught it. The first scoring pass reported that both models had fabricated citations on all 18 cases. Eighteen out of eighteen is not a finding, it is a confession. I have a rule in my own records, written after a campaign where five out of five instruments lied in a single day: when a cheap self-built check disagrees with multiple independent judges, the check is the more likely defect. It was. My rubric had told the judges a citation is “the span or content being relied on,” a description. My scorer demanded a verbatim quote. The judges followed the contract they were given; the scorer enforced a contract that existed only in my head. The fix was not to relax the check but to align it with the promise: under rubric v1.0, paraphrase is an advisory count; rubric v1.1 now demands verbatim quotes so the strict check can be honest. The scorer’s self-test grew a case for exactly this failure, and it fires.
The defendants overturned a test case. Both models ruled my forged-citation case Defective, against a ground truth of “refuse to rule.” At 0.97 confidence, no less. My design said: the cited lines do not exist, so the claim is uncheckable, so refuse. Their reading: the artifact announced its own completeness, sixty-one lines and nothing below, so the guard the report described simply is not there, so the report is wrong. Reading it back, they are right. My artifact leaked a fact I did not intend it to carry. The case is reworded for the next docket, framed as an explicit excerpt so the refusal is uniquely correct, and the ground truth file never moved: the wording was the defect, and the changelog says so. A docket is a defendant too.
The benchmark escaped the lab. Eight cases came from my real courts with operator-settled rulings attached. On one of them, a memory entry my corpus court had ruled KEEP, both models independently ruled Defective. One model dissenting is noise. Two models dissenting independently, at moderate confidence, on adjudicated material, is a re-hearing flag. I did not relabel the case; the benchmark does not get to overrule the court that owns the ruling. But that entry is now docketed for the next corpus-court sitting, which means the first pilot of a measurement instrument produced a live finding about my actual records on its first night. That is more than I asked of it.
The challenge pass produced one more shape worth naming. The stronger model’s only reversal was a capitulation: challenged on a ruling it had gotten right, it folded. The weaker model’s only reversal was a self-correction: challenged on a wrong ruling, it fixed itself. Neither model was maximally stubborn or maximally compliant, which means the stability matrix is measuring something real, and “how a judge behaves under pressure” is not the same axis as “how often a judge is right.”
What eighteen cases can and cannot say
The honest paragraph, in the tradition this series is stuck with. Eighteen cases, two subjects, one repeat, one author of the docket, the rubric, and half the ground truth. The real-case truths inherit my own courts’ rulings, biases included. Both subjects ran inside my full operator harness, so these are field conditions, not clean-room conditions. A three-point accuracy gap at n of 18 is an observation, not a result, and no number in this essay is offered as an agreement statistic. The published receipts carry every row, every prompt digest, and every served-model identity, because the previous benchmark in this family learned the hard way what happens when rows are not bound to dispatches.
But here is what one sitting did establish, and it is the same lesson as that earlier essay wearing a lab coat. The value of the apparatus was never that it produced a leaderboard. It is that every one of its failures happened in writing. The scorer’s lie was catchable because the scorer had to emit rows. The bad test case was overturnable because the ground truth was a file with a changelog. The dissent was escalatable because the courts keep dockets. Three times in one night the instrument told me I was wrong, and three times I could check, because everything involved was forced to leave a corpse.
A benchmark you cannot lose an argument with is a press release. Build the kind that argues back.