# The Subtraction Dividend

The status line was lying.

For weeks, a single command summarized the health of a personal retrieval system. Type `engraph status`, and it printed, among other things, `Model: all-MiniLM-L6-v2`. That string was load-bearing. It shaped every decision about how to make search better: which knobs to turn, which upgrades to chase, which ceiling to accept as physics. And it was false. The string was hardcoded into the status command and had gone stale. The live index was actually running EmbeddingGemma-300M, a substantially larger and newer model. The dashboard reported one engine; a different, better one was doing the work.

I lead with that detail because it's the whole essay in miniature. The thing everyone trusted to tell them what was going on was quietly wrong, and the fix that actually moved quality came not from adding a better model (the better model was already there) but from taking things away. It's a story about a retrieval system that climbed from 33% to 92% accuracy. Every single rung of that climb was a removal. Not one was an addition.

In accreted retrieval and AI pipelines full of unmeasured inherited parts, quality is often won by removing things before adding more. Files, features, components, whole categories of "intelligence": when the system is flying on assumptions, the lever that works is usually the minus sign.

## Act One: the files

The system in question is a curated knowledge vault, a few hundred hand-written notes on top of a much larger pile of machine-collected material: bookmarks, scraped papers, reference dumps. Semantic search ran across the whole thing. The benchmark was honest and small: 24 golden queries, each phrased the way a human actually asks, each with a known correct answer buried somewhere in the vault. The metric was hit@3: did the right note land in the top three results.

The whole-vault index scored 33% hit@3. Two out of three queries failed. The diagnosis, when someone finally looked at the ranked output, was not subtle: "Corpus bulk consistently out-ranked exact-match curated wiki pages. The `uv` lesson sat at collapsed rank 48, beneath 43 corpus pages." The hand-written note that answered the question exactly was sitting at rank 48, buried under forty-three pages of machine-collected sludge that merely resembled it.

So they took the sludge out of the candidate pool. Exclude the corpus directory, keep the same index, the same model, the same everything else. The score jumped to 63% hit@3. The note in the ADR is almost gleeful: "8 of the failing cases recovered the instant corpus was out of the way." Eight failures, fixed by deletion. Nothing was tuned. Nothing was added. A category of content was removed from contention and eight right answers floated to the surface.

Then they pulled two more files: the vault's root hub pages, `index.md` and `log.md`, sprawling link-farms that matched everything and meant nothing. Score: 67% hit@3.

And here Act One ends on a confident, wrong note. The ADR concluded that "67% hit@3 is the achieved ceiling with the current toolchain." It had a culprit: the model. A subsection titled "Model upgrade attempted and BLOCKED" recorded that swapping in a stronger embedding model was impossible because the indexing engine ignored the config knob. The story, as written, was: we removed what we could, we hit the wall, and the wall is the model, which we can't change. A reasonable conclusion. It was also completely false, and it stayed in the record for two days.

## Act Two: the feature

The thing that falsified Act One's ceiling was not a better model. It was discovering that a feature everyone trusted was actively making things worse.

The system had a component called, with no irony at the time, "intelligence." It was a reranker, a second-stage model that takes the initial search results and reorders them, supposedly surfacing the best ones. Reranking is a respected technique. It's the kind of thing you add to a retrieval pipeline when you want it to be smarter. Nobody questioned it; instinct doesn't point at the component with the smart name.

The 67% ceiling, it turned out, was not the model. As the second ADR puts it flatly: "67% was not a model ceiling; it was the reranker sabotaging good results." The first-stage search was finding the right notes. The reranker was then confidently shuffling them back down.

So they turned intelligence off. Removed the feature entirely. Here is the ladder, each row a single subtraction, measured the same way:

| step | hit@3 | hit@5 | MRR |
|---|---|---|---|
| baseline, intelligence ON | 67% | 71% | 0.568 |
| + intelligence OFF | 83% | 88% | 0.767 |
| + exclude `raw/` | 88% | 92% | 0.819 |
| + exclude `wiki/maps/` + `operations/` | 92% | 100% | 0.854 |
| + exclude `wiki/synthesis.md` | 92% (hit@1 88%) | 100% | 0.915 |

Turning off the "smart" feature was the single biggest gain on the board: sixteen points of hit@3 from deleting the component that existed to improve hit@3. The rest of the ladder is more of Act One: pull the raw directory, pull the auto-generated map and operations layers, pull a synthesis file that turned out to be noise. The headline, verbatim from the record: "67% → 92% hit@3, 71% → 100% hit@5 (transient peak) / 96% (durable)."

## The climax: a better component still loses

You could object here, and you should: maybe the reranker was just a bad reranker. Maybe the lesson is "use a good one," and the minus sign is incidental.

They tested exactly that. Someone configured Qwen3-Reranker-4B (a serious model, 2.9 gigabytes, in a different weight class entirely from whatever had been running) and re-ran the benchmark with intelligence back on, now powered by the strong reranker. This is the addition hypothesis in its purest form: don't remove the component, upgrade it, and let quality arrive by mass.

It lost. With the strong reranker engaged, the system scored `67/75/75/0.717` (hit@1 / hit@3 / hit@5 / MRR); with intelligence off on the same scoped config, `83/92/96/0.885`; the record's verdict was "17 points worse on hit@3" for the expensive upgrade. And the reason is the cleanest illustration of the thesis I know. The "intelligence" feature didn't only rerank; it bundled a query-expansion step *ahead* of the reranker, and that step rewrote the vocabulary-mismatched queries and dropped the right pages out of the candidate set before the reranker ever saw them. A reranker only reorders what's already been fetched. No reranker, however strong, can recover a document that expansion has already thrown away.

One experiment remains unrun: an unbundled pass that keeps query expansion off while testing the reranker alone. So the measured claim is narrower than "rerankers are bad." It is that this bundled intelligence feature, in this pipeline, was negative weight, and upgrading its reranker did not rescue the upstream damage.

That is the spine of the whole argument in one fact. Adding the strongest version of a component lost to removing the component. Not "a weak reranker hurt"; a strong one did too, and it lost upstream of its own strength. The category itself was negative weight.

## When subtraction is the right lever

Two patterns generalize out of this.

The first is **curation beats volume**, and it beats volume more violently than people expect. The 33% system wasn't missing information; it had too much. The right answer was always in the index; it was outvoted. Adding a stronger model to a polluted candidate pool just makes the system more confidently wrong, because it gets better at finding strong matches inside the noise. The corpus didn't fail because it was low-quality, exactly. It failed because its bulk gave it the votes. In retrieval, every irrelevant-but-similar document you index is a small tax on every query you'll ever run. Volume is not free. It is a liability you pay down by deletion.

The second is harder, and it's the one I'd tattoo somewhere: **an unverified feature carries negative weight until proven otherwise.** The reranker was assumed to help because it was the kind of thing that helps. It had a smart name. It used a model. It sat in the "intelligence" slot. None of that was evidence. The only evidence was the benchmark, and the benchmark said it was sabotage. A feature you have not measured is not neutral; it's a coin flip you're treating as a guarantee, and in a pipeline of several such coins, the odds that all of them are net-positive are worse than you think. The lying status line is part of this same disease: a system that cannot accurately tell you what it is running is a system whose every "improvement" is unfalsifiable. You cannot add your way out of not knowing. You can only strip back to what you can verify.

Subtraction is the right lever, then, precisely when you suspect you're flying on unmeasured assumptions: accreted content, inherited components, dashboards you trust by habit. The move is to remove and re-measure, one thing at a time, and let the benchmark adjudicate. It's unglamorous. It ships no new capability. It's also what took this system from 33% to 92%.

## The honest ceiling

I owe you the limitations, because overselling this would betray its own thesis.

92% isn't 100%. The 100% hit@5 in that table is, by the record's own admission, a "transient peak"; the durable, stable number is closer to 96% hit@5, and the ADR says so in the same breath it reports the peak. The benchmark is 24 golden queries: enough to catch a reranker sabotaging eight cases, nowhere near enough to certify the system against the real distribution of questions it'll face. This is an n=1 eval on one person's vault. The two-act climb is real and the numbers are verified, but the confidence interval around "92%" is wide, and anyone who quotes the 100% as a durable result is doing exactly the thing this essay warns against: trusting a peak reading over a measured floor.

That caveat doesn't weaken the argument. It's the argument. The discipline here is to distrust the flattering number, find the durable one, and let measurement (not instinct, not a smart-sounding component, not a stale status line) decide what stays. In systems like this one, what measurement often tells you is that something has to go.

The dividend is paid in subtraction. You just have to be willing to count.

## Postscript, 2026-07-11

The eval had one more unverified narrator in it, and I only found him this month. A performance teardown of the same retrieval system proved the ranker was nondeterministic: identical query, identical index, and the unpatched binary disagreed with itself on six runs out of ten, because tie-heavy scores were sorted in hash-map order. A stable tiebreak fixed it, ten out of ten reproducible. The direction of every number above survives. Their single-digit precision does not deserve your full trust, because the instrument that measured them could not repeat itself. I am leaving the table as published and telling you this instead, which is the whole essay again in one move: distrust the flattering number, and say so where the reader can see it.