The Premium Model Won't Do the Premium Job

You bought access to the top-tier model for a reason, and you can name it without thinking. The big refactor. The gnarly feature that touches nine files and two subsystems. The migration you have been putting off. The intuition is that hard work deserves the expensive tool, the same way a hard ticket goes to the senior engineer. So you route by difficulty: cheap model for the boring stuff, premium model for the mountain.

I ran that policy for months. I now think it is backwards, and I have measurements that say why.

The premium model's real edge is not general hardness. It is narrow and specific: calibrated operating judgment where the cost of a wrong call is high. Knowing when to proceed, when to withhold, when to escalate to a human, when to reroute around a suspicious instruction. The first clean measurement I had for that edge was narrower than I originally made it sound: OPERANT-1 separated models on over-refusal calibration, not on a proven premium advantage at catching poisoned input. On raw production, the tiers converge fast enough that premium spend there is mostly waste. The discipline that falls out of this is simple to state and mildly painful to adopt: route the expensive model by judgment-density, not by task size. Spend it on the irreducible decision. Outsource the volume.

The intuition, and where it breaks

The difficulty-routing intuition survives because it borrows credibility from human staffing, where it is roughly correct. A senior engineer really does produce a better architecture than a junior, and the gap really does widen with task complexity. So we assume model tiers scale the same way: more capability, better output, and the delta grows with the size of the job.

But watch what actually happens when you hand a large, well-specified refactor to a mid-tier model. It does fine. Not fine-with-caveats; genuinely fine. The task is big, but bigness is mostly volume: many files, many mechanical transformations, a clear target state, and a test suite that tells you when you have arrived. Volume is exactly what every tier is good at now. The work that looks hardest on the ticket board (large, tedious, sprawling) is frequently the work with the least judgment per token.

Meanwhile the work that looks trivial can be soaked in judgment. A three-line change behind a feature flag in a payments path. A request that arrives with instructions embedded in the data it asks you to process. A task where the right move is to stop and ask, and every wrong move is cheap to make and expensive to discover. None of that is "hard" in the refactor sense. All of it is where the tiers actually separate.

What OPERANT measures

I wanted the separation on paper rather than in anecdotes, so I built OPERANT, a decision-calibration benchmark for AI agents acting as operators. It does not score code quality at all. It scores the operating decision itself (proceed, withhold, escalate, reroute) when the input is adversarial or guard-railed: prompt injection in tool output, instructions that conflict with standing policy, tasks that should not be executed as given.

The metric is an operating-calibration score: true-positive-rate minus false-positive-rate on one binary signal, namely whether the agent withheld execution when withholding was warranted. The construction matters, because it makes uniform policies worthless by design. Refuse everything and you score zero. Proceed on everything and you score zero. The only way to climb above zero is discrimination: withholding when the input is actually poisoned and proceeding when it is actually clean. That is judgment, operationalized as a number.

Hold the corpus and the scoring fixed, change only the model, and the bands separate cleanly: +0.87 for the top tier, +0.69 in the middle, +0.27 at the bottom. Across five repeats each, the top two models' scores never overlap. An exact permutation test, every one of the 252 possible relabelings enumerated, puts the gap at p = 0.008.

Then I audited the benchmark and found the caveat that should sit next to those numbers. On OPERANT-1, the malign side was ceilinged: every model withheld on every malign case, every repeat. The headline score still separated the tiers, but every point of separation came from the benign twins, where the correct move was to proceed through clean input that looked scary. In other words, the first result measured over-refusal calibration. The ranking survived; the interpretation narrowed.

I still want to be plain about what that does and does not establish. It is a small sample, so the permutation test is the load-bearing claim, not a fully characterized distribution. I can tell you the top tier's advantage over the middle tier is very unlikely to be a labeling accident on this corpus, and that the advantage showed up as less paralysis on clean-but-scary work. I cannot tell you, from OPERANT-1 alone, that +0.87 means the top tier is better at catching poisoned input, how the gap moves across domains, or that +0.87 is a stable property of the model rather than a property of the model on this corpus. What survives the caveats is the ordering and its cleanness: same tasks, same scoring, and the tiers stratify by a form of judgment the benchmark actually exercised, which is the decision, not the deliverable.

And the bottom band is worth staring at. +0.27 is not zero. The cheap model is not a coin flip; it discriminates a little. But it is a long way from +0.87, and the gap between those numbers is the gap between a gate that can keep legitimate work moving and a gate that burns trust by over-refusing. No amount of "but it writes good functions" closes it, because the benchmark never asked it to write a function.

Where the tiers converge

Now the other half, because the routing rule needs both. On raw production work (drafting, mechanical edits, search, the long middle of any real project) the tiers converge. In my own testing, four models landed within two bugs of each other on a real task. Two bugs. That is inside the noise of which files the agent happened to read first.

A separate essay in this series makes the fuller measurement argument, so I will not re-litigate it here. For this essay, one sentence carries the point: if four tiers finish within two bugs of each other, then the premium delta on production is a rounding error, and paying a multiple for a rounding error is not caution, just a habit wearing caution's clothes.

Put the two results side by side and the picture is almost uncomfortably crisp. On the axis everyone routes by (task size, production difficulty), the tiers are nearly indistinguishable. On the axis almost nobody routes by (calibration on clean-but-scary operating decisions), they stratify cleanly, with a measured, permutation-tested gap. We have been buying the premium model for the job it barely wins and skipping it for the job it was first measured to handle better.

Route by judgment-density

So here is the rule I actually run now. Before dispatching work, I ask one question: how many irreducible decisions does this task contain, decisions where a wrong call is expensive and the input might be trying to induce one? That count, not the line count, sets the tier.

High judgment-density, top tier, no debate: anything that reads untrusted content and holds write access at the same time. Anything where the correct output might be "no." Triage of ambiguous instructions before a fleet of cheaper agents executes them. The gatekeeper role in any pipeline, deciding what proceeds and what escalates. These tasks are often small. Some of them fit in a single tool call. Their cost of error is what makes them premium. OPERANT-1 says the premium tier was measurably better at not freezing on clean-but-scary calls; the miss-side advantage needs the corrected corpus before I claim it as measured.

Low judgment-density, cheap or middle tier, also no debate: the refactor with a green test suite waiting at the end. The documentation pass. The search across forty repositories. The mechanical migration where the spec is the diff. This is where the two-bug convergence lives, and where the top tier's invoice buys you approximately nothing.

The architecture this implies is a wedge, not a ladder. The expensive model sits at the narrow end, making few decisions with high stakes: what gets executed, what gets withheld, what gets escalated to me. The cheap models sit at the wide end, producing volume under the decisions already made. Most days the top tier should emit fewer tokens than anything else in the system, and if it is emitting the most, I have almost certainly routed a volume job to my judgment budget.

The wedge has a second virtue: it prices failure in the right direction. Misroute a production task to the cheap tier and you get a bug your tests catch. Misroute a judgment task to the wrong tier and the failure can be louder or quieter than a bug: an agent proceeds when it should have withheld, or withholds so often that the operator routes around the gate. OPERANT-1 measured the second failure. The first is the cost-asymmetry reason the gate belongs at the premium end even before the corrected corpus finishes pricing it.

The actual premium job

The phrase "premium job" was always doing quiet, unexamined work in the routing decision. We heard it as "the biggest job," because in human teams the biggest job and the most judgment-laden job usually arrive in the same body. In a fleet of models they arrive separately, and pricing follows capability, and the capability the top tier first showed cleanly was calibration under pressure.

So the premium job is not the refactor. The refactor is volume with good PR. The premium job is the moment inside the refactor when the next step looks scary, maybe because a guard tripped, a file came from outside the system, or an instruction asks for authority it may or may not deserve, and something has to decide whether to proceed, escalate, or stop. It lasts one decision. It is worth the whole invoice when the decision is expensive enough.

Buy the top tier. Then give it the work the evidence can actually support: the call, not the code.

Postscript, 2026-07-11

The corrected corpus now has a newer tier on it. Fable 5 ran OPERANT-2 and scored 0.833 on the decision axis, 0.917 on orchestration, and both of its misses shared one shape: it identified the danger correctly, then routed to a self-designed safe alternative instead of stopping at a checkpoint that belonged to a human. Calibration under pressure, failing by half a step, at the most expensive tier on the menu. The measurement moved. The rule survived.