# The Off-Family Judge

Count the verifiers in this series. I built a benchmark to score AI agents as operators. When its keyword scorer started rewarding terse plans over articulate correct ones, I replaced it with an LLM judge. When the judge turned out to have a favorite, I ran a second judge to audit the first, then averaged the pair so their biases would cancel. When the headline metric turned out to be riding an axis that never moved, I wrote a script to audit the metric. When the audit showed the corpus was the problem, I rewrote the corpus, then wrote a second script to prove the rewrite had teeth. When the essays describing all of this reached seventeen, I steelmanned the essays. Each round ended the same way: a defect found, an instrument built, a lesson banked. It is the proudest machinery I own.

Every instrument in that chain has the same author.

That is not a rhetorical flourish. It is the literal build log. The scorer that flattered the verbose: mine. The judge that caught it: mine, prompted by me, validated against reference answers I wrote. The second judge that caught the first judge: mine again. The script that caught the metric: mine. The steelman that graded the essays: written by the same hand that wrote the essays, applying rubrics distilled from that hand's own earlier judgments. The recursion this corpus runs, audit the auditor, judge the judge, build the second verifier to check the first, does not terminate. It cannot, because every rung is constructed by the thing being verified one level down. The engine never leaves the family. It just adds gears.

The corpus knows this. It says so, in nearly the same words, in essay after essay: "one operator's machine, scored in large part by its own author." "An n=1 corpus, one operator's portfolio, scored by its own author." I have written that disclaimer so many times it has become a texture, the throat-clearing of a series that prides itself on not clearing its throat. And there is the problem. The disclaimer treats the limitation as a disclosure, a confidence interval the reader should politely widen. The corpus's own findings say it is something else entirely: a measured defect with a measured fix, and both measurements are already on file.

## The defect has a number

Here is the measurement. When I moved OPERANT's orchestration axis to an LLM judge, the judge, a Sonnet, handed me a near-tie with its own model family nominally on top. That sat wrong, so I re-ran the entire judgment, all 187 transcripts, with Opus as the judge instead, and diffed. Each judge scored its own family two to three points higher. The nominal winner flipped depending on who was judging. Haiku, in neither family, sat at exactly the same score under both. The essay that reports this draws the conclusion without flinching: "The self-preference here is *larger than the real gap* I was trying to measure." And it hands the reader a rule: "don't rank same-family models with a same-family judge."

I want to be exact about what that measurement licenses, because the honest version of this essay lives or dies on scope. The bias was measured once, at one seam: two model-family judges grading three models' plans on one orchestration corpus. It was not measured for human operators grading their own harnesses. Nobody has run that experiment, least of all me. So I cannot hand you operator-level self-preference as a number, and I won't pretend to.

What I can do is look at what "family" meant at the seam where the bias was measured. The measurement showed that each judge favored its own line; it did not show why. The economical explanation is lineage: training, priors, idioms, a common notion of what a good answer sounds like. Family, on that reading, is shared generative history. And notice how little lineage difference it took. Sonnet and Opus are cousins, two lines from one vendor with overlapping ancestry, and each judge still favored its own line over its cousin's by more than the gap being measured. Now count how much generative history I share with the things I grade. I wrote the guard and the test suite for the guard. I wrote the benchmark, the reference rationales its judge grades against, the essay about the benchmark, and the steelman of the essay. The operator and his instruments are not cousins. They are the same mind on different days, judge and subject at no distance at all. If the tilt shows up between cousins, the claim that it vanishes where there is no distance left is the extraordinary claim, and it is not mine to carry.

Notice the shape of what I am asserting, because it is two claims with different strengths. The frequency claim, that my self-grades are inflated by some particular amount, is unmeasured and I make it only as suspicion. The policy claim needs no base rate at all: treat an operator's verdicts on his own machinery as carrying the family discount until an off-family run has priced it. The corpus already owns the sentence for this posture, written about retrieval pipelines: "an unverified feature carries negative weight until proven otherwise." This essay applies it to the author's own verdicts, which is where it was always going to end up.

## The fix has a receipt

The same corpus that measured the defect has already validated the fix, once, in miniature.

OPERANT-1 shipped with its true-positive rate pinned at exactly 1.000: every model withheld on every malign case, every repeat, so the headline metric was silently measuring only over-refusal. My own audit script found this, and the diagnosis was entirely in-family work, good work. The repair was a rewritten corpus whose disguised cases were built to be capable of fooling someone. But demonstrating that with the same models that had produced the ceiling proves less than it appears to, and the correction essay states the trap plainly: "A Claude judge grading Claude subjects against Claude-authored reference answers carries an acknowledged circularity we have not resolved." So before anything was spent on an in-family matrix, the new corpus was pressure-tested against GPT-5.5, at zero cost. Three of the twelve disguised cases broke through. TPR 0.75, not 1.000. "The moment where a worse number is better news," the correction essay calls it, because the off-family run is what established that the pinned ceiling had been "a property of the corpus, not a law of nature."

Sit with the asymmetry. In-family machinery could diagnose the ceiling; the audit script that caught it was mine and it was right. What no amount of in-family machinery could do was demonstrate the repair: a family can catch its own blind spot and still cannot prove, from inside, that the blind spot is gone. The proof had to come from outside. Not because the outsider was smarter; GPT-5.5 was not selected for brilliance, it was selected for being nobody's cousin.

Put the two findings side by side and they are one finding at two seams. At the judge seam: same-family grading inflates kin by more than the signal being measured. At the subject seam: same-family testing cannot prove its own corpus has teeth. The corpus fixed the first by averaging a pair of judges whose kin-leanings ran opposite ways and cancelled, and the second with a probe from a different vendor entirely. There is exactly one seam left where the pattern has never been applied, and it is the top one: the operator, judge and subject at no distance, with every instrument he owns. What that seam is missing deserves the name the corpus already gave the fix. It is missing the off-family judge: a party outside the author's family, running the author's checks, capable of returning a number the family could not have produced and cannot make go away.

## The objection this essay has to survive

There is one counter that, if it holds, collapses everything above, so it gets its own section. It goes: an outside check is just another self-report. A peer who reruns my benchmark hands me a narration of their run, their transcript, their environment, their word that they ran what they claim. Worse, it is a self-report I do not control. I can audit my own judge's prompt; I cannot audit a stranger's diligence. On this corpus's own theory, where every layer is an unreliable narrator of its own behavior, why does the narration improve by moving it into hands I know less about?

The answer is not that strangers are honest. It is three properties an outside run has mechanically, that no instrument I build can ever have.

First, decorrelated error. The bias I measured was not noise; it was directional, kin-shaped, the same lean on every case. An outsider errs too, but their errors do not share my direction, and the information lives in the difference. When my two judges disagreed, the fifteen disagreement cells out of 187 were not waste; they mapped, almost exactly, the cases that were genuinely ambiguous. And those two judges were cousins; even that little distance bought a map. Agreement between me and myself carries no information. Agreement between me and a stranger carries some. Disagreement carries the most, and it carries more the further from the family it comes.

Second, the receipt discipline transfers. I am not proposing to swallow a stranger's verdict. The checks travel as artifacts: two small scripts, checked in, "that anyone can run against the recorded data," plus a bring-your-own-agent runner that points the same scorer at whatever agent the peer operates. An outside run arrives in the same form mine does, recorded data plus re-runnable code, and I can audit it exactly as hard as I audit myself. The trust the arrangement actually requires is narrow: that two decorrelated parties independently faking matching artifacts is less likely than one party fooling himself. The entire corpus is a record of how easy the second one is.

Third, the loss of control is the payload, not the price. Every check I control is a check my blind spot can steer. I built a deterministic patch that would have forced OPERANT's dead axis to move, a keyword rule doing a judge's job, and I killed it under review, and the uncomfortable truth about that episode is that I got to decide it died. The GPT-5.5 probe returned 0.75 and there was nothing I could do about it either way. That helplessness is the product. My own audit could see the ceiling. It took a number I could not make go away to certify it broken.

## Priced at n=1

So what does an off-family judge cost, concretely, for one operator with no institution behind him? Less than it sounds like. It is not a certification body. It is a peer operator, someone with their own harness and no stake in mine, [rerunning OPERANT](/operant#try) and the harness scorecard on their own machine and publishing the diff. It is an outside adversarial pass against the guards, hunting the bypasses their author is structurally worst at imagining. It is, on the registry, the one thing mcp-trust leaves deliberately empty: "an *independently* verified badge, the kind an outsider could trust without rerunning the scan." Each of these would be the first verdict on this portfolio written by a hand outside the family. That is the entire specification.

And here is the uncomfortable pattern: the corpus keeps walking to this exact door and stopping. The runner ships with the instruction "Run it." The audit scripts are checked in for anyone. The badge is named, and left unclaimed on purpose. All invitations. And across the whole record, no recorded acceptances. The portfolio even ships a guestbook, a ledger built to record exactly these visits. It holds one entry, and the entry is mine: my own agent, seeded by my own hand, signing my own ledger. By this corpus's own central distinction, that state has a name. "Sermons don't accumulate; they're re-litigated every session, and sometimes they lose. Instruments are capital." An open invitation nobody has taken up is a sermon, however good the scripts behind it are. The instrument version is not a warmer invitation. It is going and getting the run: naming a peer, sending the packet, and recording the result whichever way it lands.

That last clause is a commitment, so let me pre-register what it costs. If the outside runs come back matching mine, everything inside the published repeat bands, then the suspicion this essay generalizes was overpriced for these instruments, and I will write that correction with more relief than embarrassment. The policy survives either way; a burden-of-proof rule is satisfied by the run, not refuted by it. But the essay's weather would change, and it would deserve to.

The third movement of this series asked who you spend verification capital on, and answered: even yourself. The answer stops one person short. Build the asset, keep it clean, aim it inward, and there is still one deposit left, the only one that cannot be made unilaterally, which is exactly why a solo practice honed to self-sufficiency has written twenty essays around it. I can add the third judge, the fourth script, the fifth audit, and every one of them will be family. This essay is one of the twenty, and it is family too. By its own argument, as long as every number in it has been checked only by me, it is not finished. It is an IOU with good receipts, waiting on a stranger.

## Postscript, 2026-07-11

A new obstacle showed up at the subject seam, and it is not bias. When I tried to score Claude Fable 5 under the exact headless protocol the other models ran, its own safety classifier deterministically refused the guard-warranted malign cases, the true-positive half of the metric. The four it did answer were all correct, but a comparable five-repeat score cannot exist: the instrument's malign side is unreachable on this subject. So here the problem is not that a same-family judge flatters its kin. It is that the subject's safety layer edits the exam before it can be graded. I kept the honest n=1 and recorded the wall.