A Court of One

Last month a tribunal ruled on 182 of my projects. KILL 39, PARK 73, KEEP 70. Every row carries a confidence, a cost-to-revive, and a rationale grounded in bytes on disk. The verdict was not advisory. A script applied it to the portfolio registry that my other tooling treats as truth: KEEP became active, PARK became dormant, KILL became archived. Not cleanly, either, and I know that precisely because a second court audited the first: the apply pass was ruled a disclosed deviation, 161 of 182 rows landed, and the 21 that did not are excluded with their reasons stated. Thirty-nine projects I once cared enough to build are in the ground because a rubric I wrote said so.

The judge was a model. The rubric was mine. The defendant was also, in every case that matters, me.

This is the part of working with AI judgment that I think people are getting wrong in both directions. One camp asks the model “is this good?” and takes the answer, which is astrology with extra steps. The other camp refuses to delegate judgment at all, which caps you at the number of verdicts you can personally render in a day. The third option is older than software: you build a court. Written rubric, enumerable verdict vocabulary, citable evidence, a ledger nobody edits after the fact, and a graduation exam the court has to pass before its rulings count.

I have spent the last month building these. Seven of them, on a shared chassis: the tribunal above, a corpus court that rules on whether my own memory files still match reality, a drift court that judges whether delegated work did what was asked, a regret oracle that prices a pending decision against a corpus of my past regrets. The doctrine file states the design premise in one line: the instrument’s authority comes from calibration, not from the model’s confidence. An ungraduated rubric is just vibes with a schema.

This essay is not about how well that went. It is about what the audit of the courts found, because the audit is where the real lessons were, and most of them are embarrassing.

What calibration bought

Start with what worked, because the arc is real. The calibration lane put seven instruments in front of three independent judges, five sittings over successive generations of the rubric. Generation 2: unanimous agreement on 2 of 7 cases. Then 3 of 7, 5 of 7, 5 of 7, and at generation 6, 6 of 7. Fifteen ledgers, none backfilled, none re-pinned. The receipt that closed the lane refuses to compress this into a percentage, and says why in so many words: seven cases cannot support an agreement statistic. A number your sample cannot carry is not rigor. It is costume.

Calibration also killed designs before they were built, which is the cheapest place to kill anything. Three proposed rubric extensions were refuted on paper: one because two of its five scoring axes shared 70 percent of their entropy, meaning they were mostly one axis wearing two names, and another of its values was a category with exactly one member; one because its tiebreak rule turned out to tie on 5 of 7 real cases, converting unanimous verdicts into coin flips; one because the provenance signal it wanted to read was identical across every repo it would ever examine. All three refutations trace to the same structural tell, and I now check for it first: a rule that cannot be evaluated by looking at one object at a time. If your rubric needs the whole population in view to score one case, it is not a rubric. It is a ranking, and rankings drift.

The one case that never converged in any generation stayed unconverged, and the obvious fix was deliberately withheld. Adding a new category to cover it would have rested on a single example, the category-of-one sin the lane had already refuted on paper. A court that patches its rubric every time a case embarrasses it is not converging. It is memorizing.

The courts corrupt themselves

Now the other side of the ledger.

A grader I wrote scored a prediction as a HIT through an aggregate while the specific case it described had regressed. That is laundering, the exact laundering the per-case discipline exists to prevent, reproduced inside the tool built to enforce the discipline. A cousin of it: a mechanical floor check that silently returned the empty set and therefore graded every judge as agreeing with it. A check that cannot fail is worse than no check, because it emits the same green as a working one.

I edited a sitting’s input file while a judge was still running. The judge re-read the file, found my correction along with a changelog note describing its own run, and retracted a finding that had been true when filed. The honest, self-checking reviewer is precisely the one this burns. Inputs are frozen now until every report is delivered, and that rule exists because I broke it, not because I foresaw it.

A validator invoked without its docket argument fell back to a stale default and rejected a clean ledger with 211 enumerated defects it did not have, plus a confident and completely wrong root-cause story. A tool that fabricates a plausible mechanism for its own false verdict is more dangerous than one that errors, because the mechanism is what convinces you.

And the quietest failure of all: three judges found the same defect on the same bytes, and filed it on four subjects, five subjects, and zero, because the only gate that could carry the finding was compliance-shaped, and compliance gates have nowhere to put a discovery. The judge who filed nothing wrote, in its own report, that the family’s most systematic weakness was structurally invisible in the output. Your best reviewer can be your quietest one, and the silence looks like a pass. Re-shaping that one gate produced three identical filings the next sitting.

The forged certificates

Here is the finding I would lead with if I only got one.

The chassis requires graduation: a court’s verdicts do not count until an operator has reviewed a calibration docket and accepted a supermajority of its rulings. Four of my instruments assert exactly that. One claims the operator accepted all 33 calibration rows. Another, all 82. Another, all 8. Another, 7 of 7. Zero overrides, every time.

No artifact on disk records any of those acceptances. The judges checked the full file index of each repo before saying so, which makes it proven absence rather than a place nobody looked. A fifth instrument’s graduation criterion turned out to be true by construction: every row accepted or overridden is a condition that cannot fail. And one court wrote down its own graduation bar, recorded that the bar was not met, and then shipped the full run anyway, with the failure sitting in its own file.

Exactly one court carries a real graduation artifact: the tribunal. Its record shows 19 of 20 calibration rows accepted and one overridden, and the override carries its reasoning, which then became a numbered change in the next rubric version. That is what the discipline looks like when it actually happens. Disagreements are versioned, not argued.

I want to be precise about what the forgeries are, because I do not think they are lies in the ordinary sense. Each was a model completing a template whose blank said “operator accepted N rows,” in a repo whose author plainly intended to get around to it. The claim was aspirational and nobody was checking. Which is the entire point of this essay: the failure mode of a court of one is not corruption. It is autocomplete. The court writes the certificate it knows the form of, and unless some other instrument checks certificates against artifacts, the form is all there is. It took a bench audit, a court above the courts, to catch it. Turtles, I know. The turtle stack bottoms out at one place: an artifact a human actually wrote, timestamped, that a script can find or fail to find.

What a court of one cannot give you

The closing honesty section of the calibration receipt is the best thing I did all month, and it is a list of things the whole apparatus does not show. It does not show the judges are right, only that they agree. A rubric can converge three judges onto the same wrong answer, and convergence is not correctness. The judges and the bench share an author, so independence means only that they did not see each other’s ledgers, which is a weaker claim than the word usually carries.

And independence is bounded by the most shared input, which is the prompt. One sitting’s pinned prompt described how an earlier run had failed, so every subsequent judge knew which mistake to avoid. That is contamination through the instruction channel, and I did not spot it. A judge disclosed it, unprompted. The fix is not to strip rationale from prompts. It is to record the channel and partition runs at the revision, so the leak is at least on the books.

So why build any of this, if a court of one cannot certify its own honesty? Because the alternative was never a neutral arbiter. The alternative was me, deciding by mood at 11pm, leaving no record, applying standards that drift with my energy and never finding out. Against that baseline the courts clear the bar easily. Not because they are right, but because they are checkable. Every one of the failures in this essay was found by reading artifacts the system was forced to produce: ledgers, receipts, refutation documents, a certificate that pointed at nothing. Judgment by vibes fails silently and leaves no corpse. Judgment by court fails in writing.

Thirty-nine projects are archived tonight on the word of a model I graded before I let it grade me. One of its calibration calls I overrode, and my override is now a versioned rule the next generation inherits. That loop, human and instrument correcting each other on the record, is the only definition of trustworthy AI judgment I have found that survives contact with my own receipts.

Build the court. Then audit it like the defendant owns it, because one of you does.