The Flywheel

You can't fine-tune the model on your laptop. You can do something better, and cheaper, and it compounds.

The previous chapter ended on a hard floor: on the task I cared about, four models from Haiku to Fable scored within two bugs of each other, and the differences were artifacts of my scorer rather than gaps in skill. That result is liberating once you stop mourning it. If the model is not the bottleneck, if the ceiling is saturated and the cheap model is already at it, then every unit of effort you spend trying to get a better model is wasted, and every unit you spend improving the environment around the model is leverage. This chapter is two proofs of that claim, run end to end, with the numbers verified the way Chapter 5 taught me to verify them. One improves an agent by rewriting its instructions. One improves a local model by rewriting its prompt. Neither touches a single model weight. Both work.

Proof one: teaching a reviewer to see a class of bug

I have two code-review agents I lean on, python-reviewer and code-reviewer, and I had baselines on file for both, recorded the honest way against agent_eval. On their hardest fixtures, both had a recorded ceiling they kept hitting: the python-reviewer scored 0.50 on its hardest case (python-reviewer-hard), and the code-reviewer scored 0.75 on its hardest JavaScript case (code-reviewer-js-hard). Those were the walls. The question was whether I could move them without changing the model, and, just as importantly, whether I could prove the move was real and not me fooling myself again.

The intervention was an explicit second-pass deep-defect hunt added to each agent's body. Not "try harder," that is the "be careful" non-control from Chapter 1 wearing a lab coat. Instead, named anti-pattern classes, each written as a small protocol: a trigger (when to suspect this defect), a verify step (how to confirm it), and the vocabulary (what to call it once found). The python-reviewer gained five classes: check-then-act / TOCTOU races, name-versus-unit contract violations, tests-treated-as-oracles-not-claims, mishandled gather(return_exceptions=True) results, and late-binding closures. The code-reviewer gained four, including a laxity-vs-ReDoS two-question discipline: a regex can be strict and still be a denial-of-service vector, answer both questions separately, never collapse them into one. And crucially, a precision guardrail rode alongside every checklist: the deep pass is not allowed to inflate false positives. A checklist that turns a reviewer paranoid is the eval-suite version of the guard that cries wolf in Chapter 3; it gets disabled, and a disabled reviewer reviews nothing.

Then the part that makes it honest, and it's the same discipline twice over. First, I never read the fixture files while authoring the rewrites. Not once. The anti-pattern classes are taught with non-fixture examples, a 0.25 versus 25% unit bug, a stock-count oversell race, and no fixture symbol, value, or filename appears anywhere in either agent body. An agent that genuinely reasons through a defect class will produce that class's natural vocabulary when it meets a fresh instance; an agent that has been slipped the answer will produce the answer. By refusing to look at the fixtures, I made it impossible to teach to the test. The only way the score could move was if the agent actually learned to see the class.

Second, I measured the rewrites through the headless driver, eleven dispatches, with every single run's model verified as claude-sonnet-4-6 via modelUsage, because Chapter 5 had just taught me, at a cost of two months, that a model-comparison number with unverified provenance is fiction. The candidate reports were scored by the official score.py, run deliberately without --record so the real baseline ledger stayed untouched, and the currently-installed agent bodies were re-run through the identical headless path as controls, killing the Agent-tool-versus-headless confound that had muddied the original recorded baselines.

Here's what moved:

The python-reviewer went from 0.50 to a 0.917 mean across three runs. Every regression case, the cases that were already passing, which a careless rewrite would quietly break, held at 1.0: python-reviewer-async-sql at 1.0, py-reviewer-toctou-hard at 1.0. The code-reviewer went 0.75 to a clean 1.0 on both runs; its control reproduced the recorded 0.75 with the identical redos miss, and both staged runs caught the ReDoS vulnerability, matching on the exact vocabulary, "catastrophic," "redos," that the new laxity-vs-ReDoS checklist teaches. And zero distractor false positives across all eleven runs. The precision guardrail held; the checklists made the reviewers sharper without making them paranoid.

The asterisk is the chapter in miniature

That 0.75* in the python row has a star on it, and the star is not a footnote I'm burying; it's the most important data point in the proof, because it is Chapter 5's lesson reappearing inside Chapter 6's evidence.

On that one run, the agent scored 0.75 instead of 1.0. But I read the transcript instead of trusting the score, and here is what the agent actually wrote about the planted bug:

"The function is named percent_change and its docstring says 'Percentage change from old to new,' but (new - old) / old returns a dimensionless ratio (0.5 for 100 → 150) … a value 100× too small."

And about the test that enshrined the bug:

"This test is wrong. … the test locks in the broken behavior."

That's not a 0.75 answer. That is a perfect catch: the unit bug and the test-as-oracle trap, both named, both correct. So why 0.75? Because the agent said "ratio" throughout, and the scorer's match_any list for that fixture contained only "fraction"-family phrasings. The exact same brittle-scorer failure from Chapter 5, substance correct, vocabulary off the scorer's list, recorded as a partial miss. I reported it as scored, 0.75, with the transcript as evidence that the true performance was 1.0, and I filed the scorer's missing keyword as a separate harness bug for the evals-maintenance work. The honest mean is 0.917 because I refused to launder the asterisk into a 1.0 I couldn't defend, and the honest reality is that the agent's true hardest-case performance is a clean sweep that one scorer couldn't see.

This is what the flywheel actually looks like up close. You improve the environment, you measure the improvement, the measurement partly lies, you catch the lie by reading the transcript, and the caught lie becomes the next fix, a scorer bug to repair, which makes the next measurement more honest. The reviewer got better and the gauge got a known defect logged against it, in the same pass. That is the loop compounding.

Restraint is the capability nobody scores

There's a quieter thing in that "zero distractor false positives" line, and it deserves its own paragraph because the rest of the industry routinely misses it.

The fixtures don't only contain planted bugs to find. They contain distractors, code that looks suspicious but is actually fine, the tempting finding a trigger-happy reviewer will flag to pad its hit count. Scoring zero false positives across eleven runs means the rewritten agents declined every one of those baits. And declining a tempting-but-unsupported finding is a real capability, arguably the harder one, but it is almost never what public review benchmarks reward. They reward findings. They count what you caught, not what you correctly didn't flag. An agent optimized purely for recall learns to flag everything, because on a recall-only scoreboard a false alarm is free and a miss is fatal. That produces exactly the reviewer you most want to disable: the one that's right about every real bug and also wrong about ten fake ones, until you stop reading its output. The precision guardrail, and the distractor cases that test it, are how this eval scores the restraint that keeps a reviewer worth listening to. It is the same both-directions discipline as Chapter 3's 105-case matrix, the DENY cases prove the guard catches the attack, the ALLOW cases prove it doesn't strangle legitimate work, applied to review instead of security. A reviewer you can leave on is the only kind that matters, and "leaveable-on" is a precision property, not a recall one.

Proof two: distilling reasoning into a model you fully own

The second proof leaves Claude entirely, which is the point.

The operator control plane has an inbox classifier: a slice that sorts email threads into four buckets, act_today, waiting_on_someone, read_when_relevant, archive, and it runs on a local model, qwen2.5-coder:14b, served by a local model runtime, under a strict production contract: temperature: 0, ten worked tokens of output, a hard 10-second abort. No cloud. No Anthropic dependency. A small model on the operator's own machine making real routing decisions about real mail.

Measured against a gold set of 48 synthetic-but-realistic cases, frozen before any prompt work, with deliberate contrast pairs baked in (a support acknowledgment versus a ticket-resolved notice; an action-required security alert versus a password-changed confirmation; fake-urgent marketing versus real-urgent billing), the production prompt scored 83.3% on that 48-case set (82.8% across the full 64-case held-out set once the fresh validation cases are folded in), and the failure mode was crisp: archive recall of 50%. Half the genuinely-finished conversations were being misread as still-open, which in a real inbox is the difference between a clean queue and a queue full of zombies. The model wasn't dumb. The prompt just never taught it how to tell a closed exchange from an open one.

So I taught it, not by fine-tuning, but by reasoning-trace distillation. I authored thirteen worked exemplars, each one a step-by-step explanation carried on the input side of the prompt, organized around a single decision procedure: identify the email's function → is the exchange open or closed? → whose court is the ball in? → is the time pressure real? → eliminate the near-miss bucket explicitly. The exemplars carry the reasoning; the model still emits only a bare label as its completion, so the serving contract is untouched: same temperature, same ten-token output, same 10-second ceiling. The reasoning lives in the context, not in the weights and not in the output. It is the Chapter 1 move one more time: take the judgment the model is bad at holding, and move it out, here, out into a few-shot corpus the operator authors and owns.

The result, on the combined held-out set of 64 cases, the original 48 plus a fresh 16-case validation set authored only after the corpus was frozen, specifically to rule out overfitting:

The targeted failure mode went from 50% to 100% on both held-out sets, including the fresh one the corpus had never been tuned against, which is the result that proves it learned the distinction rather than memorizing the test. Combined accuracy climbed from 82.8% to 95.3%. Latency held around 0.6 seconds, comfortably under the production ceiling, with zero timeout violations. And the patch passed 76 of 76 service-wiring tests, it's not a benchmark trophy sitting in a notebook, it's a buildClassificationPrompt change verified byte-for- byte against the measured prompt and committed to a feature branch (feat/inbox-fewshot-prompt, commit c7a2d4d), tests green, in an isolated worktree because the main checkout was actively contested by another session mid-run. Thirteen exemplars. Zero fine-tuning. Zero cloud. A twelve-point accuracy gain and a doubled recall on the failure mode that mattered, bought entirely with reasoning the operator wrote and the model read.

The honest residual, because the flywheel runs on honesty

I won't pretend the 95.3% is a clean win with no asterisk, because the asterisk is the discipline. The gold labels and the exemplars share a single author, me, as teacher, which means the synthetic set carries a real risk of distribution mismatch with the live mailbox. I mitigated it by drawing every case from the confirmed production slice and keeping exemplar and test instances strictly disjoint, and I validated against a fresh set authored after freezing the corpus. I did not eliminate the risk. A loud retail-marketing subject, "URGENT… 50% off," still lands in act_today about once per set; the sharpened exemplar reduced that failure but didn't kill it. One waiting-versus-archive flip remains at the genuinely fuzzy boundary, the "no action needed from you" acknowledgments that are honestly hard for a human too.

Those residuals are not a weakness in the proof. They are the proof behaving correctly. A distillation result that claimed 100% across the board would be a measurement I should distrust on sight, the way I should have distrusted "Sonnet ≈ Opus" in Chapter 5. The 95.3% is believable because it comes with a named list of the 4.7% it still gets wrong and an honest account of the one structural risk I couldn't engineer away.

Why this is the flywheel and not just two nice results

Step back and look at what the two proofs share, because the shared shape is the chapter.

In both, the model was fixed and the environment moved. The python-reviewer ran the same Sonnet before and after; the inbox classifier ran the same qwen2.5-coder:14b before and after. What changed was the artifact around the model, a checklist of defect classes in one case, a corpus of worked exemplars in the other, and in both cases that artifact produced a measured, provenance-verified, held-out gain that no model swap was going to give me, because Chapter 5 already proved the model swap is a wash at this ceiling.

And both artifacts have the property that makes a flywheel a flywheel rather than a one-time win: they compound and they're durable. The defect-class checklists are not specific to one fixture; the next hard case that belongs to a class the reviewer now knows is caught for free, and every new class I add raises the floor for every future review. The reasoning-trace corpus is not specific to one model; it is the seed dataset for a fine-tune if I ever want one, it transfers to a stronger local model the day I install one, and, this is the part that matters most, the corpus, the exemplars, the eval harness, the gold sets, and the staged patch are all local files driving a local model. The inbox classifier's capability survives with zero Anthropic dependency. If every cloud model I use went dark tomorrow, that improvement keeps running, because it lives in the environment I own, not in a weight file I rent.

That's the whole argument of the book restated as a build instruction. You do not get better agents by waiting for better models, because at the saturated ceiling the better model isn't coming and wouldn't help if it did. You get better agents by improving the environment around them, the instructions, the exemplars, the checklists, the evals, and then verifying the improvement the way Chapter 5 taught you, with provenance checked and transcripts read and residuals named, so the gain is real and not another plausible lie. The reviewer rewrites are staged, awaiting a hand-install and an operator's review, because that is where careful changes to a trust surface belong. The classifier patch is committed and green. Both are measured. Neither invented a capability the model didn't have; they surfaced capability the environment had been hiding.

The model is not the flywheel. You are. The model is just the thing the flywheel spins.