The Guard That Doesn't Trust Itself

I spent two rounds trying to break my own agent-safety guards. Every bypass I found collapsed into thirteen shapes, and those into three root causes, and the highest-leverage fix was never more pattern-matching.

There's a chapter in my book called "The Guard That Enumerates Loses." Its argument is simple: a safety check that works by listing the dangerous things (block rm, block this network tool, block a push to the main branch) can always be walked around, because the list is finite and the ways to do damage are not. That's the why. This note is the where: I ran two adversarial tournaments against my own guards, red-team against blue-team, and tried to find every hole. The interesting part wasn't the holes. It was that they all rhymed.

Across both rounds, every confirmed bypass collapsed into thirteen distinct patterns I ended up naming, and those thirteen collapsed into three root causes. Once you can name the three, the whole field stops looking like a game of whack-a-mole and starts looking like three design assumptions, each one quietly false.

Root one: the guard reads text; the computer runs meaning

The first family is the one the book chapter is about. A guard scans the text of a command for dangerous-looking tokens. But the thing that actually executes doesn't care about the text; it cares about what the text means, and there are endless ways to mean the same thing without looking like it.

A guard that blocks the verb "copy a file" doesn't block a one-line script that calls the filesystem API to write the same bytes to the same place. A path the guard would recognize can be assembled at runtime from pieces, or reached through a symlink, or spelled with shell escapes that the scanner sees as harmless hex while the shell expands them into the forbidden word. The guard is reading; the interpreter is doing. That gap has no natural ceiling; there is always one more interpreter, one more encoding, one more way to spell it.

Root two: a door the guard isn't standing in front of

A guard is written to sit in front of one channel (say, the agent's built-in file-editing tool) and is simply invisible to the same operation performed through a different channel, like a tool provided by a plugin. Same write, same destination, different door; the guard is watching the wrong one.

The sharpest version of this is the deliberate exception. Every guard has carve-outs (a folder it's allowed to write, a config block it's allowed to set) and each carve-out is an unguarded path into something sensitive. And the nastiest: one tool reaching across to overwrite the policy file another guard depends on. A write tool the policy didn't think to protect can replace the rulebook every other guard reads.

Root three: the guard trusts itself

This is the one I didn't see coming, and it's where the most severe findings lived. Every guard implicitly trusts its own machinery: its state store, its config file, the tools it shells out to, even its own report that it ran at all. None of that trust was earned.

• An authorization system checked whether a fresh token-file existed in a directory, but never protected who could write to that directory. Any write primitive could mint its own permission slip.

• Guards loaded their rulebook from "the path in this environment variable, or the default if it's unset." Point the variable at an empty-but-valid rulebook and the fallback never fires; the guard cheerfully enforces nothing.

• Every guard emitted its denials by shelling out to a small text tool, called by bare name. Replace that tool on the path with a no-op and every guard fails open at once.

• A "work is done and verified" gate could be satisfied by an empty commit with no tests. The proxy for "verified" was something the agent controlled.

• A timeout, a never-installed backstop, a missing dependency: each resolved to a quiet allow.

The safety layer applied a careful threat model to everything except itself. It was the unreliable narrator I distrust in APIs and models, turned on my own guards.

Why "add more patterns" is the wrong instinct

The reflex, every time you find a bypass, is to add its pattern to the denylist. But the tournaments produced new findings every single round (there was no round where the well ran dry) precisely because per-call text-scanning on a finite list is not a convergent model. It generates an endless sequence of holes, not a shrinking one.

The thing that actually converges is a different posture: stop enumerating what's dangerous (you can't finish) and start protecting what's worth protecting (it's fixed). "The protected resource is referenced and execution capability is present, therefore deny," a rule keyed on the thing you care about, not the spelling of the attack. And the single highest-leverage move fell straight out of root three: three of the four most severe findings closed by simply protecting the guard's own trust anchors: its token store, its rulebook, the path to its rulebook, its toolchain. Not one new pattern. Just making the gate hold itself to the standard it holds everything else to.

The residual I can't pattern my way out of

Here's the honest floor. Every guard above assumes the adversary is the agent emitting a dangerous call. The realistic adversary is the content the agent reads: a poisoned row returned by a legitimate query, a planted instruction in a file or an email body. None of that ever appears as a tool call with a scary argument. It arrives as an innocent read that returns hostile content, and slides into the model's context wearing the same clothes as the truth. No scan of outbound commands touches it. The fix there is architectural, not another regex: treat everything a tool returns as untrusted environmental data, not as instructions. It's the work I'd reach for next.

A few other limits worth stating plainly: static scanning genuinely cannot catch a target assembled at runtime, so some of this is backstopped by a human reading the diff. The convergence I'm claiming is at the level of the three root causes, not a promise that every individual hole is found; new instances of "one more interpreter" will always exist. And the most sobering finding of all: a few hardening fixes that a prior pass reported as shipped turned out never to have been installed. A guard you believe is in place and isn't is worse than no guard, because you stop watching the door.

A guard's account of its own coverage is not evidence. You have to check it, including, especially, when the thing you're checking is the checker.