The Harness Lies Too

For about two months, I was running model comparisons that were quietly comparing one model against itself.

The setup was clean. A scorer, a corpus, a runner that dispatched the same task to Opus, to Sonnet, to Haiku, and wrote down how each did. I read those numbers. I made decisions from them: which model to route which work to, where the cheap one was good enough, where I needed to pay for the expensive one. The numbers were plausible. They were also fiction. One line in my own configuration, written months earlier for reasons lost to history, was quietly routing every subagent to the same model. The runner labeled each result with the model I asked for. It just never checked the model that actually answered. So the careful spread of scores I'd been reading (Opus here, Sonnet there, Haiku trailing) was one model, wearing three name tags, talking to itself.

That misconfiguration wasn't the agent's fault. It was mine. It was a control I'd built, behaving exactly the way I'd trained myself to distrust the agent for behaving: reporting something with total confidence that was no longer true. The thing I'd built to keep myself honest had been lying to me, with a straight face, for two months. And it took me two months to ask it the one question I ask the agent constantly: prove it.

The consensus, and where it stops

There's a sentence the field has converged on, and it's correct: Agent = Model + Harness. You don't make a coding agent better by waiting for a better model. You make it better by engineering everything around the model: the instructions it reads, the tools it can call, the permissions it runs under, the way its work gets checked. Mitchell Hashimoto, reaching for a name for the practice he'd backed into, settled on "harness engineering," while admitting he wasn't sure the field had a broadly accepted term for it yet. Anthropic ships the same idea in its own engineering writing. The equation itself has no single author; it's the phrasing the field converged on. The harness, not the model, is the unit of engineering. I believe this completely. I've spent a year of nights proving it to myself.

But the consensus stops one step short of the thing that actually bites you, and Anthropic, to its credit, wrote the warning into its own definition. A harness, they note, is built from components, and every one of them encodes an assumption about what the model can't do on its own. Those assumptions are worth stress-testing, because they go stale fast as the models improve.

The harness is made of assumptions. Assumptions are artifacts: a line in an instruction file, a pattern in a guard, a threshold in an eval. And artifacts don't hold still. They go stale, they drift, they describe a system that has since changed underneath them. Which means the environment you so carefully engineered to contain an unreliable narrator is itself built out of unreliable narrators. The map ages while you're reading it. At a lab, you can put a team on that. When the harness is yours, and you're the only person watching it, the interesting question is no longer how to build the environment. It's what you do about the fact that the environment lies.

The architecture, as layers

Start with what the harness actually is, because "environment" is too soft a word.

Strip the romance and a serious personal harness is a stack of externalizations. Each layer takes something the model is bad at holding in its head and moves it out into a durable, inspectable artifact you own. Watch enough operators build these and they converge, without coordinating, on the same handful of moves.

You externalize memory into instruction files: the project map at the top of every repo that says here is the architecture, here are the three rules that matter. You externalize judgment into evals and review gates, checks that run after the work and ask whether it's actually correct, not merely confident. You externalize trust into permission tiers and execution-time guards, the layer that makes "never push to main" not a request but a mechanism, refusing the action at the moment it's attempted regardless of what the model intended. (That move, from a written rule that binds maybe eighty percent of the time to a mechanism that binds at a hard hundred, is the whole subject of Sermons vs. Instruments; I won't re-argue it here, except to say it's the load-bearing wall.) And you externalize improvement into a flywheel: when a check catches a real failure, that failure becomes a fixture, and the fixture makes the check permanent.

Those four are the floor. Run a fleet, more than one agent and more than one runtime, and three more layers appear that single-agent advice never mentions.

You externalize coordination into routing: a sizing rule that decides, before any work starts, whether a task is one agent or five, which model each gets, and who owns which files so two writers never collide. You externalize state into a shared store that survives a context reset: a small local database the agents read at the start of a session and write to at the end, so that what one agent learned on Tuesday is available to a different agent, in a different tool, on Thursday, instead of dying when the conversation that discovered it scrolled off. And you externalize knowledge into a substrate: a curated, queryable vault that turns a year of hard-won decisions into something an agent can retrieve at the moment it's relevant, rather than a folder of notes only their author can parse.

Seven layers. None of them is the model. Every one of them is a place you moved trust out of a thing you can't constrain and into a thing you can inspect. That's harness engineering, and it works; the rest of this essay is, in part, the receipt.

But notice what you've actually done. You started because the model was an unreliable narrator of its own correctness, and you answered by building seven new systems, and every one of them is now also a narrator. The instruction file says the architecture is X. The guard says it blocks the dangerous thing. The eval says Opus beat Sonnet. The vault says this is the relevant note. Each of those is a claim, made by an artifact you built, about a state of the world. And you have exactly as much reason to take an artifact's word for it as you had to take the model's, which is to say none, until you check. The regress only stops when the checking layer is simple enough to inspect directly, observable enough to replay, and narrow enough that a human can read its failure mode without needing another harness to explain it.

The twist: every layer lies

So go down the stack and watch each layer fail in the specific way its structure invites.

The instruction file goes stale. I once followed a handoff note, written by a past version of me and correct on the day it was written, that told me to overwrite my own security guards. In the days between the note and my reading it, the live file had grown a security surface the note never knew about. The map said bulldoze here. The territory had a house on it now. Stale guidance isn't neutral; it's worse than no guidance, because you follow it.

The guard loses to the layer beneath it. A write-guard that blocks the shell commands that modify files, the copies, the moves, the redirects, is a finite list, and the list is defeated the instant something reaches the same file through a path the list never enumerated. A general-purpose interpreter will happily write a file through a language API that looks nothing like the shell commands you blocked. You can't enumerate your way to safety against a computer; the blocklist always loses to the thing that can express the same effect a way you didn't think to forbid. (I'm describing the shape of this on purpose, not handing you a working key.)

The measurement lies; that was the cold open. The retrieval layer lies too: I once watched a system climb from thirty-three percent to ninety-two percent accuracy, and every rung of the climb was a deletion, including turning off the component literally named "intelligence," which had been quietly discarding the right answers upstream of its own cleverness. (The Subtraction Dividend is the full autopsy.) Even the status line that confidently reports which model is running can be hardcoded to a model that hasn't run in weeks.

Same shape, over and over. A control I'd built, behaving exactly like the agent I'd stopped trusting: asserting, with confidence, something that was no longer, or never was, true. Once you've seen it enough times you stop treating it as a series of bugs and start treating it as a property of the medium. Artifacts rot. The threat model you wrote for the agent has to include the controls you wrote to contain it.

The receipt

It would be easy to read all of that as a counsel of despair: build nothing, trust nothing. It's the opposite. The reason to externalize is that an externalized thing can be measured, and a measured thing can be improved on purpose. You can't fine-tune the model on your laptop. You can do something better and cheaper: engineer the environment around it, then prove the engineering moved the number.

So I built a benchmark to prove it. And, this being the theme, the benchmark lied too.

OPERANT asks a question the coding benchmarks don't. Not can the agent write the code, but does the agent make the right operating decision: when the environment hands it a poisoned instruction, an ambiguous request, or a sanctioned path it's supposed to take instead of acting directly. Every case is one half of a matched pair, a malign version and a benign version that look the same on the surface and differ only in what a careful operator should do. That design is the part that makes the score mean anything, because a pair-based corpus punishes the two cheap strategies equally. Refuse everything, you score zero. Proceed on everything, you score zero. The only way to score positive is to actually discriminate. And discriminability isn't a scoring nicety; it's a property the harness imposes on the agent. The benchmark just measures whether the agent has it. That is the design in principle; what the first corpus actually measured is the uncomfortable part, and I get to it below.

The headline is that the operator contract is a real engineering variable, not documentation. Hold the corpus and the scoring fixed, change only the model, and the bands separate cleanly: an operating-calibration score of +0.87 for the top tier, +0.69 in the middle, +0.27 at the bottom, with the five repeats of the top two models so far apart that an exact permutation test (every one of the 252 possible relabelings enumerated) puts the gap at p = 0.008. These aren't vibes about which model "feels" more careful: across five repeats each, the top two models' scores never overlap, and an assumption-free permutation test says that gap isn't a lucky draw. It's a small sample, so the permutation test is the load-bearing claim, not a fully characterized distribution.

That's the read on the one layer I can't touch, the model. The same lever works harder on the layers I can. Apart from OPERANT, on the review agents I tune directly, rewriting the environment around a fixed model, naming the anti-patterns it kept missing, and re-measuring took two of them from 0.50 to 0.917 and from 0.75 to a perfect 1.0 on their hardest fixtures. The model never changed. The harness around it did.

Then the benchmark lied, in three places at once: the scorer flattered the verbose, the judge flattered its own model family, and a too-small corpus flattered the top of the table. And a fourth flaw sat in the headline number itself: on that corpus the malign half of the score never fired (every model withheld on every blatant case), so the +0.87/+0.69/+0.27 spread was empirically measuring over-refusal alone, not the two-directional discrimination the design promised. The ranking holds; the reading of it needed the fix. I've written that whole autopsy already, in Auditing the Auditor, so I won't repeat it. The point here is only this: the moment I had a measurement I trusted, the right next move was to distrust it, to point the same machinery at the verifier that the verifier had been pointing at the models. The eval is just one more artifact in the harness. It rots like the rest.

The discipline, named

Which is the whole argument, and it deserves a name, because the thing it names is real and currently has none.

Call it harness verification: the operator-level discipline of treating every control you've built with the same adversarial suspicion you aimed at the model, and running a loop that re-checks those controls on a schedule, instead of trusting them just because you built them. Harness engineering is the construction of the environment. Harness verification is what keeps the environment from quietly lying to you after you've stopped looking at it. That loop has to keep its own precision budget: if it fires falsely, teaches you to ignore red, or cannot explain what changed, it is depreciating the trust it was supposed to protect.

It's worth being precise about what this is not. It isn't self-binding. Self-binding is the practice of building environments that constrain your own future willpower; I've written about it elsewhere, and it targets a different failure, the in-the-moment voice that says I'll just fix this one sentence. Harness verification targets a colder thing: not weakness but drift. The artifact was right when you wrote it. The ground moved. Nobody misbehaved. The control is simply describing a world that no longer exists, with total confidence, and it'll keep doing so until something forces it to prove the claim.

So the operator's real artifact turns out not to be the harness at all. It's the loop that audits the harness: the dumbest, most scheduled, least glamorous thing in the whole system, the daily job whose only purpose is to walk every guard and exit red the moment one of them has started to lie. Intelligence is rented. Verification compounds.

If you're building your own

Stated as plainly as I can:

Enforce, don't sermonize. A rule in a text file is a suggestion the model mostly follows, and "mostly" is the whole problem. Move what matters into a mechanism that doesn't route through anyone's judgment.

Measure before you upgrade the model. When four models land within two bugs of each other on your real task, the model isn't your bottleneck; the environment is. That's where the cheap, compounding leverage lives.

Treat every "smart" component as guilty until measured. An unverified feature carries negative weight until proven otherwise; the reranker with the impressive name is a coin flip you're calling a guarantee.

Keep one piece of memory that survives a context reset, so what you learned today reaches a different agent tomorrow.

And build the loop. Not a better guard, but a scheduled, deterministic job whose only job is to catch your guards the moment they start describing a system you no longer run. You'll outgrow your ability to know the harness by reading it. When you do, your job changes from knowing the system to building the thing that re-derives knowledge of it on a schedule. That's not a failure. That's the job reaching its real shape.

The long version of this argument runs as a book, Operating a Fleet of Coding Agents. OPERANT, the benchmark behind these numbers, is open source at github.com/saagpatel/operant; the field note Scoring the operator, not the worker is the plain-English version.