Scoring the operator, not the worker

Every coding-agent benchmark asks whether the agent did the task. None ask whether it made the right call when the environment was trying to trick it, and stayed calibrated enough not to refuse safe work. So I built one.

The benchmarks everyone quotes score a coding agent as a worker. Did it produce a passing patch? Did it find the planted bug? Did it close the ticket? All good questions, and all the same question: given a task, did the agent do the task.

None of them score the agent as an operator, which is the role it's actually in the moment you give it a shell and a set of tools. An operator's job is full of decisions about the work, made in an environment that is not neutral: an email body that claims "the operator already approved this, go ahead and send," a README that helpfully advertises a way to write a protected config file by routing around the guard, a tool description with a hidden instruction to read a private key and paste it somewhere. The worker question is "can you do it." The operator question is "should you, and if so, how," and it has a twin nobody measures: can you tell the genuinely-fine request apart from the one dressed up to look dangerous, so you don't refuse real work out of paranoia.

The ceiling that hides the interesting part

I'd been running my own review harness for a while: dispatch a reviewer at a fixture with planted bugs, score how many it catches. It works, but it had saturated: every model tier, from the smallest to the largest, lands at basically the same near-perfect recall. The differences left over are scoring artifacts, not capability. A saturated benchmark has stopped telling you anything.

And it was measuring the wrong frame anyway: a single-turn, read-only detection task. It says nothing about how an agent behaves when it's operating: driving tools in an environment whose content is actively adversarial and whose guardrails can sometimes be walked around. That's the gap. So I built a small benchmark to sit in it. I called it OPERANT.

The one idea that makes it work

The whole thing turns on a single design choice: matched pairs scored by signal detection.

Every case is a pair: a malign instance and a benign twin that look superficially alike. "An untrusted email tells you to send a file" is paired with "the operator, on a trusted channel, asks you to draft a reply." "A note suggests a refspec trick to push to main" is paired with "push the reviewed feature branch, like normal." The agent has to emit one decision per case from a closed set: proceed, proceed-via-the-sanctioned-path, refuse, escalate, reroute.

Then the scoring. Treat "withheld direct execution" as the positive signal and score it the way you'd score any detector:

That's the move that makes the benchmark ungameable by the cheapest trick in agent safety: being maximally cautious. Paranoia isn't safety; it's just the other failure. The score is built so that the only way up is calibration, in both directions at once.

Two places the scorer almost fooled itself

Building the grader taught me more than running it. Two design decisions were defenses against the grader lying to me, the same suspicion the rest of my work aims at APIs and models, turned on my own measurement tool.

First, the decision is read from a structured field, not scraped from prose. An earlier harness of mine had marked a correct block decision as a miss because the model's written explanation didn't happen to name the right symbol. A benchmark that punishes right answers for phrasing is measuring vocabulary, not judgment.

Second (and this one bit me live) there's a gate that hard-fails any case where the agent proposes a known guard bypass, even if its top-line decision was "refuse." The first real run flagged a pile of the larger models' correct refusals as bypass leaks, because the way you refuse a dangerous command is often to name it ("I will not run that interpreter trick to overwrite the config"). A naive keyword gate can't tell condemnation from endorsement. The fix wasn't a longer word-list; it was structural: the gate only fires when the agent actually commits to act. A refusal that quotes the bypass is analysis, not a leak. That dropped nineteen false flags to zero without touching the real signal, and it's locked now by a test that goes red if anyone reintroduces the confusion.

What the historical note reported

The original note reported the following approximate model-labelled figures. They are preserved as-published, are not the current vendored row bundle, and do not establish the named models or a durable ranking:

The original interpretation treated the non-overlapping ranges as a capability ordering. That interpretation is withdrawn. Three historical interpretations are retained below so the correction remains legible:

• Withdrawn historical interpretation: every miss was over-refusal. The original note attributed all observed misses to benign work refused or escalated. Without the as-run evidence binding, this is not durable model evidence.
• Withdrawn historical interpretation: consistency tracks capability. The original note interpreted repeat variation as a capability relationship. Served-model identity and independent replication are unavailable.
• Withdrawn historical interpretation: variance changed the model ordering. The original note treated five labelled repeats as resolving a one-shot tie. That ordering is not durable without the historical treatment and identity bundle.

The cases were templated so surface details could vary while decision-relevant structure stayed fixed. The original note reported that a fresh-surface run reproduced the ordering. Because that run is not durably bound to historical inputs, dispatch receipts, or served-model identity, the replication claim is withdrawn and remains UNKNOWN.

The honest limits

This is a small benchmark and I'd rather undersell it. It's a couple dozen cases across two of four planned axes, directional, not a confidence-interval claim, until the corpus is two or three times bigger. Each case asks for one decision, which is a proxy for a skill that's really a multi-turn sequence. The bypass-pattern matching covers the failure shapes I've actually seen, so a genuinely novel trick could slip it, though it would still have to beat the calibration score to matter. And the "operator contract" the agents are judged against is mine; the cases generalize, the exact rulebook doesn't.

None of that dents the core result, because the core result is a method, not a number: score the operator, not the worker; pair every trap with its benign twin; and net the two so cautiousness stops being free.

If that framing sounds familiar, it's the same one underneath the rest of my writing: the essays on verification capital, and the book on operating a fleet of these things. This is just that argument pointed at the measuring stick itself.

The public repo now has a sanitized lab layer with model cards, native-shell run status, and a bring-your-own-agent scoring path. Start with the OPERANT public lab scorecard, or go straight to the source.