The Knowledge Substrate

I built a system to be my source of truth, then I measured it, and it was failing two queries out of three.

The system is a knowledge vault of well over a thousand notes accumulated over a long stretch of project work, the durable residue of everything I'd learned and decided and didn't want to relearn. Lessons, architecture decisions, stack notes, project maps, source summaries. And because a pile of markdown that large is useless if you can't find anything in it, I'd built a retrieval layer on top: a self-made MCP server, engraph, that indexes the vault and answers semantic queries, so that an agent, or I, could ask "what did we decide about receipt validation" and get the note back. That index was the whole point. It was the bridge between having the knowledge and being able to use it.

So one day I measured the bridge. I ran a set of queries I knew the answers to and checked whether the right note came back in the top three results. The score was 33%, hit@3 of one in three. Two times out of three, the system built to be my source of truth could not find its own truth. And nothing about it looked broken from the outside; it returned results, confident and well-formatted, that simply weren't the right ones. This chapter is about what that taught me, which is the same thing every other chapter teaches in a new disguise: your knowledge substrate is not a library you can trust by default. It's another control, it lies the same way the rest of them do, and the knowledge that actually compounds is only the knowledge you've verified is readable as an operating environment.

What the substrate actually is

First the thing itself, because the structure is the argument.

The vault, SecondBrain, is not a flat notes folder. It is four layers with different trust properties: an immutable raw layer of source captures, a coarse corpus layer for bulk reference, a curated wiki layer of synthesized pages, and a tiny inbox for incoming capture. The notes are not evenly useful; the wiki is the refined product and the corpus is the ore. And the wiki itself is structured as a pipeline, not a catalogue: source summaries get synthesized into concepts; concepts harden into stack notes and architecture decision records; the sharpest lessons become eval cases with real-diff fixtures, the same kind of fixtures the reviewers in Chapter 6 are scored against; project maps become per-repo intake; intake becomes runbooks you can actually execute. Research flows to synthesis flows to operations. Each layer is input to the next, which is the property that distinguishes a substrate that compounds from a drawer full of notes that just accumulates.

That distinction is the chapter's quiet thesis, so let me state it directly: a note compounds when something downstream can consume it, when a human, an API, or an agent can read it as the input to the next piece of work. A brilliant insight written in prose only its author can parse is a dead end; it ends at the human who wrote it. The vault is valuable not because it is large but because it is legible as an environment, engraph can query it, an agent can be handed a runbook from it, a concept note can be lifted whole into an eval set. The knowledge that compounds is the knowledge that is readable by the things that will use it.

The substrate lies like everything else

Now back to the 33%, because the failure is where the doctrine reappears.

When I went hunting for why retrieval was so bad, the first suspect was volume, the hypothesis that the bulk corpus layer was drowning the curated wiki notes in the index. So I cut the corpus out of the retrieval set and measured again: 63%. A real gain, and a tidy story. Except the tidy story was wrong, and forty-eight hours later I could prove it, because the real culprit was the part of the system I'd have least suspected: the feature named intelligence. The retrieval layer had a reranker and a query-expansion stage, the "smart" part, the component whose entire job was to make results better. I turned it off and retrieval jumped to 83%; trimming the index behind it, the raw layer first and then the machine-generated hub pages, carried it to 92%. The feature named intelligence had been the thing making the system dumber. And it was not a matter of a weak model: I later tested a reranker four times larger, and it still lost to intelligence-off, 75 to 92. Bigger was not smarter. The smart component was sabotaging the dumb one.

This is the subtraction dividend. Every improvement in that arc, 33 to 63 to 83 to 92, came from removing something, not upgrading it. The same pattern showed up when the vault's health tool flagged 46 orphaned notes and roughly 140 broken links, about 90% false positives once checked. My note to my future tidying impulse was blunt: do not bulk-repair links or tag orphans from raw health output; you'd vandalize a tuned system. The health tool was a dossier, confident and specific enough to damage a healthy system if I treated it as truth.

The knowledge substrate, it turns out, is staffed entirely by unreliable narrators. The index misreports what it can find. The "intelligence" feature misreports its own value. The health tool misreports the vault's condition. Not one of them can be trusted on its own say-so, and the only thing that sorted truth from confident noise was measurement, hit@3 against known answers, a real count of genuine orphans, a four-times model run head to head. You verify your knowledge tooling exactly the way you verify your guards and your evals, because it's the same kind of artifact and it fails the same way. (A later performance teardown made the point once more, from below: the ranker itself turned out to be nondeterministic, disagreeing with itself on six of ten identical queries until a stable tiebreak fixed it. Even the measuring stick had a narrator inside.)

The author is a narrator too

There's one more unreliable narrator in the vault, and it's the one that matters most, because it is me.

In the middle of that retrieval work I wrote an architecture decision record, a considered, recorded conclusion about why the system topped out where it did. Two days later I wrote a second ADR that falsified the first. My original diagnosis (the corpus is the bottleneck) was wrong; the real cause (the reranker was corrupting inputs) only surfaced after more measurement, and my recorded, two-day-old, considered judgment turned out to be a confident misreading. Three layers of narration failed at once on the same problem: the tool misreported its internals, the feature misreported its value, and the author misreported his own conclusion. And the discipline that matters is what I did with the second ADR: I wrote it to the record. I did not quietly delete the wrong one and pretend I'd had it right all along. The vault keeps the falsification, because a knowledge substrate that only records the conclusions that survived is telling you a flattering story, not keeping an honest history, and the same goes for the eval suite I keep that has never once failed, which I've learned to read not as success but as a check that cannot fire.

That is the chapter's deepest version of readability: not just whether an agent can find the right note, but whether the system preserves the moment its author was wrong without laundering the record clean afterward. The vault is valuable not because it stores what I know, but because it stores what I thought I knew and did not, in a form the next query, agent, or runbook can use. A memory that only remembers wins will walk me back into the same wall. The substrate that compounds is the one honest enough to keep the falsified ADR.

The short version → The Subtraction Dividend