The Missing Denominator

This summer I put my own error log on trial, and the bench nearly hung on a single question: when a failure stops happening, who gets the credit?

Over nine weeks, my agent harness logged 7,146 tool errors across 6,739 working sessions. Clustered by signature, they reduce to thirty recurring failure classes: a guard refusing a dangerous delete, an edit rejected because the file was never read first, a web fetch redirected into a tool that a second policy then denies. I have written about putting my repositories on trial before a judge that had to cite evidence for every verdict. This is the same chassis pointed at a new docket: three independent judges, model instances working blind to each other under a written rubric, every claim anchored to pinned bytes, every number reconciled by a validator before a ledger is accepted.

Most of the rubric converged fast. One gate refused to, sitting after sitting, and the story of why is the most useful thing the whole exercise produced.

The gate that would not settle

The gate, in plain language: before you call a failure class fixed, check whether reduced traffic explains its silence just as well as a repair does.

It exists because the corpus kept inviting exactly that mistake. A database lock-contention cluster went silent for a month, and the known fix for it was sitting on an unmerged branch the entire time. Lock contention is proportional to load. Fewer concurrent writers produce precisely the same silence as a repair. The silence was real. The fix was not in place. Anyone reading the chart would have closed the ticket.

In its first two sittings the gate was the most productive rule on the bench. On every cluster with a closed window, some judge could name a non-repair explanation: a five-day batch job ending, a workflow that ran once and was never re-run, a workload that simply moved elsewhere. The counts were stable too, six or seven fires per judge, with a spread of one across all three ledgers.

Then a judge pointed out, correctly, that the gate only looked at clusters that had gone fully silent, and said nothing about clusters that were merely declining. So the wording was broadened, from “the silence is load-confounded” to “the silence or the decline.” The next sitting came back 23, 8, 23.

Two judges fired the gate on essentially every downward-moving cluster in the corpus. One fired it on eight. Same bytes, same rubric, same gate.

A defensible reading on each side

The divergence was not sloppiness, which is what made it worth taking apart. The gate asks whether reduced load explains a decline as well as a repair does, and it never says who carries the burden of proof.

Read it wide, and the confound must be affirmatively excluded before the decline can count as improvement. The corpus contains no measure of how much work was attempted each month, so the confound can never be excluded, so the gate fires on every decline. Read it narrow, and the confound must be affirmatively plausible: fire only where you can point at a specific workload stopping. Both are honest readings of the same sentence. One unstated choice was worth fifteen fires.

The narrow judge did something genuinely clever, and it deserves to be recorded along with the reason it failed. They built a proxy denominator out of the corpus itself: for each declining cluster, check whether the segment that dominates it kept producing errors elsewhere in the same month. If the delegated-agent traffic was still erring plenty in July, they argued, then delegated-agent load did not fall, and a decline in a delegated-dominated cluster cannot be blamed on reduced load. Checked against the pinned bytes, the individual facts held up. But the aggregate error volume from that same segment fell by half over the same window, which cut the other way, and both cuts share one flaw: they use errors as the measure of exposure. Errors are the numerator. The proxy was the numerator standing in for its own denominator, circular by construction. It narrowed the question without answering it.

What the corpus was shouting

While the judges argued burden of proof, the corpus itself was carrying the strongest evidence, and one judge finally read it out loud: twenty-four of the thirty clusters peak in the same month.

A schema limit, a database lock, a file-deletion guard, a redirect hook. These failure classes share no code, no tool, no cause. Twenty-four independent repairs do not land in one four-week window. A synchronized peak across causally unrelated series is what a shared denominator moving looks like, the tide going out under every boat at once. The corpus was not describing thirty stories of fixing and breaking. It was describing one story about how much work the machine was asked to do each month, refracted through thirty error signatures.

One number

The fix was not a third rewording. We pinned an exposure measure into the corpus, total tool calls per month, alongside every cluster’s counts. And, following the bench’s standing discipline, the expected answer was registered in writing before the judges were dispatched: the gate should fire on exactly five named clusters, and the spread across judges should collapse.

All three judges returned exactly those five clusters. Spread of zero. The worked arithmetic from one ledger shows why the convergence was instant. A cluster’s raw errors fell from 187 in June to 183 in July, which reads as a modest improvement. Divide by the work: 187 errors over 69,038 tool calls, then 183 over 29,541. Per unit of work, the failure got 2.29 times worse in the month its raw count improved. Fire.

The lesson is about diagnosis, not about this gate. Narrowing the wording did nothing. Broadening it made the divergence explode. Supplying the missing measurement converged three independent judges on identical arithmetic immediately, on the first try, to the registered answer. The gate was never ambiguous. It was unanswerable, and rewording an unanswerable question only moves around where the judge has to guess.

What this changes

I now read every trend chart in my own tooling differently, and I would invite you to do the same with yours. “Errors are down this month” is a fraction with only the numerator shown. In my nine-week window, every single declining raw count was compatible with the traffic simply leaving, and in the one case we worked end to end, the decline was hiding a doubling. A repair claim without a denominator is not a weak claim. It is a mood.

The transferable test is for the moment your reviewers, human or machine, keep splitting on a judgment call. The reflex is to edit the rubric, because wording is cheap and you can always produce another draft. Spend one round on wording if you must. If calibrated judges still disagree, stop asking what the question means and start asking what measurement would make it answerable, and whether your evidence contains that measurement at all. Disagreement between honest judges is sometimes a vocabulary problem. More often than I expected, it is an instrument reporting, in the only voice it has, that you are asking it for something the data cannot say.

The limits

The bench demands this section. Convergence is a property of the instrument, not proof the verdicts are true: three judges agreeing on arithmetic means the question became computable, and this docket still has no answer key. And the denominator is itself a choice with consequences. I normalized by tool calls. Normalized by session count, which fell much harder over the same window, the rates come out different, and a cluster can improve against one measure while worsening against the other. Pinning the denominator turned a fight about burden of proof into a calculation. Choosing the denominator is still a judgment, and it is still mine.

Which is where the opening question lands. When a failure goes quiet, the credit goes to whoever can show the denominator. Nobody in my corpus could, for two full sittings, and the bench did exactly the right thing with that: it refused to convict the silence of being a fix.