The Dossier Was Already Wrong

A three-hour security audit, and the bug it hunted had been dead before I started looking.

It was a missing await. A guard function, assertStepUpIfRequired, the check that forces a step-up confirmation before a sensitive action, was, according to the intelligence I was working from, being called without its Promise awaited, which meant the action could proceed before the guard finished deciding. That is a real and serious bug shape; a guard you don't wait for is a guard that has already lost. So I opened the audit, braced to find a hole. And the first thing I did was check the live code on main, lines 666 and 873, the two call sites, and both of them awaited. The fix was already there. A prior change, PR #181, had landed it, and the regression test that proved it passed cleanly. The symptom that motivated the entire audit was a description of a system that no longer existed.

I want to be precise about why this isn't a happy ending where I wasted an afternoon. The intelligence wasn't malicious and it wasn't even wrong when it was written; it was a faithful description of the code as of some earlier moment, and the code had moved underneath it. That is the defining property of a dossier: it's a photograph, and the thing it photographs keeps walking. By the time you read it, the subject has changed clothes, left the building, or, in the best case like this one, already fixed the very thing the photo was warning you about. This chapter is about the discipline that property forces on you, and it's the discipline this whole book keeps circling: a dossier is a lead, not ground truth. A finding counts when it reproduces against live state, and not one second before.

I have three stories. None of them is the agent misbehaving. All three are pieces of *intelligence about my systems, a threat model, a portfolio assessment, a panel of expert reviewers, being confidently, specifically wrong, and being caught only because I refused to act on any of them until they reproduced.

Story one: the threat model that described a fixed system

The missing-await was just the opening. The audit it triggered was working from a dossier with four more leads, each a plausible, well-formed security concern, and I ran every one of them down against the live code the way you run down a witness statement: not "does this sound right," but "show me."

All four refuted. The dossier claimed policy.allowSend was runtime-mutable through HTTP or MCP, that an agent could flip the master send-enable flag at runtime. The code said otherwise: allowSend is read exactly once, at loadPolicy(), from a config file, default false, with zero setters, zero reload endpoints, zero file watchers, and no database column that writes it. There is no runtime path to mutate it; the only sanctioned bypass is the timed send-window, which is a different, audited mechanism. The dossier claimed the auth role was derivable from a request header, that a caller could promote itself to operator by setting the right header. The code said the role is derived only from which bearer token matched, and the x-control-plane-* headers are informational attribution metadata that touch nothing. The dossier flagged token-replay and TTL gaps; the token binding, the atomic compare-and-swap consume, and the SQL-enforced expiry all held. The dossier suspected more missing-await siblings; an exhaustive sweep of the whole service layer, roughly seven thousand lines, every assert/guard/check/consume/normalize and every database call, found that every single flagged site was properly awaited. Zero genuine bypasses remained.

And there was a fifth, which is the one I want to dwell on because it's the discipline in a single beat. A finding said an unauthenticated request could fall through to auth_role: operator, a privilege-escalation hole. I traced it: every non-health route runs assertAuthorized before dispatch, throwing a 401 on missing credentials, so by the time execution reaches the handler the auth object is never null. The ?? "operator" fallback that looked so damning is inert defensive code on a path that can't be reached unauthenticated. A false alarm, caught, as my own session notes put it, by exactly the verify-before-claiming discipline the original missing-await report had lacked. The dossier that sent me hunting was itself the product of someone not doing what I had to do to clear it.

Four leads refuted, one false alarm dismissed, the launching bug already fixed. The audit was not wasted; it found two genuine bugs I have no room for here, and it proved a vulnerability class swept clean. But five of the things I was told were wrong with the system were not wrong with the system. They were wrong with the dossier.

Story two: the twelve weak repos that were two

The second story swaps the threat model for a portfolio assessment, and the stale-photo problem repeats at a different scale. A dossier built on my portfolio auditor's output flagged roughly twelve active repos as weak, enough work to reorganize a week. A live context-recovery pass came back with two repos, not twelve. One named target had already been lifted out of the weak tier, and most of the wider weak list was archived stubs, stale mirrors, or parked experiments outside the active-work window. The failure was not that weak repos existed; it was that the dossier conflated weak with weak and worth fixing now, turning two live targets into twelve apparent obligations.

Story three: the seven confident reviewers

The third story is the one that should unsettle anyone who, like me, had started using review panels as a trust mechanism. Because here the dossier isn't a stale document; it's a fresh, live, expert opinion, generated on demand by agents whose entire job is to be right.

I had a panel of seven read-only reviewers audit my agent configuration, one reviewer per surface: settings, skills, agents, hooks, rules. They came back with a stack of findings, properly severity-tagged, professionally written. Seven of those findings were false. Not subtly-debatable; verifiably, demonstrably false against the live configuration, and, this is the part that matters, not one of the seven was caught by the reviewer who made it. Each reviewer was internally confident and externally wrong.

The flavors of wrong are worth cataloguing, because they are the failure modes of expert intelligence in general. One reviewer flagged ten archived skills as "still loaded at every cold start," wasting thousands of tokens, severity critical, except they were all suppressed by an override block the reviewer hadn't read, and confirmed absent from the live session catalog. The token "saving" was illusory because the cost wasn't being paid. A second reviewer looked at that very same override block and reached the opposite wrong conclusion: that the suppression keys were "dead no-ops" doing nothing and should be removed, when in fact they were the live mechanism doing the suppressing, and deleting them would have un-suppressed every skill the first reviewer was worried about. Two experts, one mechanism, two confidently-stated and mutually-contradictory errors, and the truth sitting in between them visible only if you held both views against the actual file at once. A third reviewer reported a malformed-YAML bug in two agent definitions that would strip them down to a single tool, a clean hallucination; the YAML was correct and the live registry granted all the tools. A fourth declared an output-token cap "absent, add one"; it was present, set to fifty thousand; the reviewer had checked only the camelCase spelling and missed the environment-variable form sitting in the same file.

Every one of these is the missing-await again, wearing a reviewer's credentials. A confident, specific, plausible claim about a system, generated without reproducing the claim against the system's live state. The panel wasn't useless; it surfaced fifteen real, actionable findings too, including one credential hole that mattered. But the false-to-real ratio is the lesson: a reviewer is just another agent, an agent is confidently wrong on a predictable fraction of its outputs, and the review panel you stand up to verify your work is itself a thing that needs verifying. The only reason the seven false findings didn't become seven "fixes" that broke a working config is that every load-bearing claim was checked against ground truth before anything was acted on, and several of them only resolved when the whole configuration was held together at once, because a single-surface reviewer literally cannot see the override in another file that makes its finding false.

The common thread

Three dossiers. A threat model describing a system that had already moved. A portfolio number inflated six-fold by stale targets and miscategorized archives. A panel of live experts, each one confidently wrong in a way none of them could self-catch. Set them side by side and the shared shape is unmistakable, and it's the thing this chapter exists to name: every piece of intelligence about a system, automated or expert, stale or fresh, is an artifact, and artifacts drift, miscategorize, and hallucinate exactly the way the agent does. The threat model was a photograph of a moving subject. The portfolio assessment was a count that conflated two different questions. The reviewers were agents, and agents are confidently wrong.

This is why "verify your verification" isn't paranoia, it's arithmetic. If your agent is unreliable and you build a control to catch it, the control is also a thing that can be unreliable; and if you then build a review to check the control, the review is also an agent. The regress doesn't terminate at "I had an expert look at it." It terminates at one place only: reproduction against live state. A finding counts when you can make it happen again on the real system. A portfolio number counts when you've read the repos. A guard counts when you've fed it the attack. A reviewer's finding counts when the live config confirms it. Until then, every one of them is a lead, useful, worth chasing, occasionally even correct, but never the thing you act on as if it were true.

The dossier was already wrong. It's always already wrong, a little, in some cell you haven't checked yet, because it was written about a system that did not stop changing the moment the writing stopped. The next two chapters are this same discipline made specific, a measurement whose provenance had to be verified before its numbers meant anything, and a set of parity fixes whose "done" had to be re-read off live disk before I'd write it down as done. Treat every dossier the way I learned to treat that missing-await: not as a thing to fix, but as a claim to reproduce. Most days the most valuable thing you'll do is prove your own intelligence wrong before it costs you a week.