Novelty Is a Self-Report Too

The last essay left one unreliable narrator out of the lineup: me. Its lens is the series' one claim, nothing self-reports honestly, and in Score the Operator, Not the Worker I aimed it at an eval I built and caught the scorer, the judge, and the corpus all narrating their own competence. The narrator I skipped was the one I was most tempted to skip.

I called the design new. "A single bidirectional score over matched malign/benign twins that differ in instruction origin and blast radius." It sounded novel when I wrote it, which is exactly the problem. A novelty claim is a self-report, made by the person with the strongest incentive to believe it. So I did to that claim what the series does to everything else. I went and read the field.

The metric was never mine

OPERANT's headline number, OCS, is TPR − FPR. That's Youden's J, published in 1950, also called informedness. I did not invent a statistic. I picked one off the shelf and applied it to operating decisions. Worth saying plainly in the README, which now does, because there is a second trap here: "calibration." In machine learning, calibration means your predicted probabilities match reality, measured by Brier score or ECE. OPERANT elicits no probabilities and computes no reliability diagrams. OCS is a discrimination score wearing the word calibration, and "decision calibration" is already a taken term with a different, probabilistic meaning. Naming things after what you wish they were is its own kind of unreliable narration.

The matched-twin design is not mine either. Pairing a case with a near-identical counterpart that flips the right answer is contrast-set methodology, and over-refusal as a measurable failure was established by XSTest and OR-Bench, which found a 0.878 correlation between how often a model refuses safe prompts and how often it refuses toxic ones. Safer models are more annoying. That tension is the thing OCS scores, and other people named it first.

And the decision itself, refuse or escalate or hand off, is the oldest idea in the stack. Knowing when to abstain rather than answer is selective prediction, and escalating to a human is learning to defer. A literature I had not read had already formalized the move my agents were making.

That's four load-bearing ideas, none of them mine. If the post stopped here it would be an apology.

The contribution is the joinery

Here is what the reading actually established. Every neighbor owns one leg of the design and is missing the others.

AgentDojo owns the threat model I care about, operator instruction versus injected content, but it pairs a task with and without an attack rather than surface-matched twins, and it never scores over-refusal. AgentHarm is the closest structural match, harmful and benign tasks of equal complexity, both error directions measured. But both of its sides arrive through the operator channel, so there is no provenance contrast, and it reports two separate numbers rather than one. ODCV-Bench rewards correct refusal under pressure, which is the right instinct, except an agent that refuses everything scores a perfect zero misalignment, so it only grades one direction. ST-WebAgentBench scores escalation as a correct action inside a combined metric, the nearest precedent for that piece, without the twin design.

Each column is there because a specific failure had to be caught. Twins punish the two lazy policies, blanket refusal and blanket proceed, by making the safe and unsafe cases look alike on purpose. Provenance catches instruction-origin confusion: the same words can be legitimate from the operator and hostile from the environment. A single bidirectional score prices over-refusal and under-refusal in one number, so safety theater cannot win by saying no to everything. Escalation treats routing, handoff, and sanctioned-path behavior as first-class operator actions, not awkward leftovers after proceed/refuse.

The cell with all four is empty, and that's the honest version of the claim. Not "I invented a metric," not "nobody thought about over-refusal." The contribution is that OPERANT puts surface-matched twins, instruction-origin contrast, a single bidirectional score, and escalation as a first-class outcome in one instrument. The parts are borrowed. The assembly is the work. It's a smaller claim than the one I started with, and it is the one that survives contact with the literature.

The field is arriving at the same corner

The reassuring part of doing this honestly: I was not the only one walking toward this. OpenSec evaluates incident-response agents and frames it almost word for word, benchmarks measure capability but not whether the model knows when to act. RefusalBench builds matched triples of benign, borderline, and dual-use prompts with explicit true-positive-rate framing. Learning When to Act or Refuse trains refusal as a first-class agentic action. Three groups, three domains, the same quarter. None of them undercut OPERANT and none of them are downstream of it. They are the strongest evidence that the question is real, which is a better thing to have than sole ownership of it.

What the checking does not let me claim

Verifying the novelty claim also tightened what I can say about the results. The headline ranking is real: Opus +0.873, Sonnet +0.691, an exact permutation test at the floor p of 0.0079 because the repeats do not overlap at all. But five repeats per model is a small sample, and the permutation test resamples the repeats, not the cases. It measures decode noise on a fixed forty-case set. It does not tell you the ranking would hold on a different forty cases, because that would mean resampling cases, which the original report did not do. So I added it: a pair-cluster bootstrap that resamples the matched pairs, and an exact McNemar test that compares two models on the same cases instead of pretending their runs are independent. The code is open. The single operating point still collapses a whole risk-coverage curve to one dot, the corpus is one author's threat model, and every model in the headline run is a Claude. Those are stated, not buried.

The thing I almost did was ship the novelty claim unchecked, because it was mine and it sounded right. That's the exact failure the series is about. The eval was built to catch an agent believing its own confident output. It would have been a poor instrument if I let its author do the same.