The Control Plane

The bug that scared me most did not crash anything. It told the truth about a lie.

In the operator control plane, the local service that gates every outward action my agents take, there is a state machine that tracks each pending action through its life. It's the thing standing between an agent's intention to send an email and an email actually leaving the building. An approval is pending, then approved, then sending, then sent. Clean, legible, the kind of state diagram you draw on a whiteboard and feel good about. And buried in the recovery path was a sequence that could turn one of those labels into a confident lie.

Here is the sequence, traced through the real code in app/src/service/review-approval.ts. When the service sends an approved draft, sendApprovedDraft writes the state "sending", then awaits the actual Gmail call, then, on success, writes "sent". Three steps. Now kill the process in the gap between step two and step three: Gmail has accepted the message, the email is genuinely on its way to a real human, but the "sent" write never lands. The approval is frozen mid-flight in "sending", and no provider message-id is persisted until the success write, and that is what makes it dangerous: there is no idempotency key, nothing to dedupe against. Fifteen minutes later a janitor sweep, recoverStaleSendingApprovals, finds the stuck record and relabels it: state="send_failed", last_error_code="send_interrupted". And then the action gate, assertApprovalActionAllowed, cheerfully permits an approve from send_failed, because of course it does, a failed send should be retryable.

Read what just happened. The state says failed. The reality is delivery unknown, the email may very well have been sent. An operator looking at "failed" does the obvious, correct-seeming thing: re-approve, re-send. And the recipient gets the same email twice. Real money if it's a billing notice, real reputation if it's a client. The state machine did not crash. It produced a label, the label was false, and a human trusting the label took an unsafe action on its authority. That's the whole chapter in one bug: a control plane's deepest job is not to act; it is to never lie about what it did, because everything downstream trusts the label more than the truth.

Gate every outward action

Step back to the principle, because the bug only matters in light of it.

An agent that can send email, open pull requests, post to channels, or call third-party APIs has outward reach, it can make things happen in the world that you cannot un-happen. Chapter 3 was about containing what an agent can do to your machine; this chapter is about containing what it can do to everyone else. And the move is the one the whole book keeps making: you don't ask the agent to be careful about sending. You put a gate between the agent and the outside world, the gate is enforced in code the agent cannot talk past, and every outward action flows through it.

In the control plane that gate is a four-step flow, and I want to walk the clean path, not just the broken one, because the architecture of the happy path is what makes the broken one fixable. An agent drafts: mail_draft_create writes a draft artifact, nothing leaves. The agent requests approval: approval_request_create mints an approval_id and moves the record to pending. Then the two steps that an agent cannot perform, approve and send, are operator-only, and each is protected by a confirmation token whose construction is the point.

The token isn't a password. It's a cryptographic binding. The digest is SHA-256(approvalId : action : draftDigest : token), which commits, in one hash, to which approval, which action (approve is not send), and which exact draft content. That binding is what makes replay impossible: a token minted to approve cannot be replayed to send, a token for one approval cannot be reused on another, and if the draft text changes after the token is minted, the draftDigest no longer matches and the token is dead. The service enforces that last property directly: mutating a draft nulls the digest and expires the approval, so there's only ever one outstanding token per approval, bound to exactly the bytes the operator saw when they decided to trust it. The token has a ten-minute TTL enforced in SQL, not in application code, and it is consumed by an atomic UPDATE-as-compare-and-swap against single-writer SQLite: single-use, no time-of-check-to-time-of-use window, no way for two sends to both believe they won the token. (The book's evidence table elsewhere ascribes a specific bit-length to that token; I could not confirm the exact entropy figure in the session sources for this chapter, so I describe the binding, which is the load-bearing security property, and leave the bit-count marked [unverified] until I read it out of the token-generation code directly.)

That's the positive path: draft, request, approve-with-bound-token, send-with-bound-token, every irreversible step requiring a fresh single-use credential the agent can neither mint nor reuse. It's a good design. And a good design is exactly the kind of thing that lulls you into not auditing it, which is how the double-send bug lived in the recovery path of an otherwise-solid state machine.

The state machine's one job

So I audited it, the way Chapters 3 and 5 taught me to audit my own controls: not by reading the code for elegance, but by reasoning adversarially about where the labels could lie.

The double-send fix is small and it is the right kind of small. The recovery sweep already had a message, "inspect Sent mail before retrying," but that message was advisory. It was a sentence printed for a human, and nothing in the state machine enforced it. (You will recognize this from Chapter 2: a human instruction is not a code contract. "Be careful" addressed to a machine is the same non-control as "be careful" addressed to an agent.) The fix turns the sentence into a transition rule: an approval marked with the send_interrupted error code, which signals delivery-unknown rather than a real Gmail rejection (real rejections carry the provider's actual error code), can no longer be silently re-approved. It must pass through the operator-only reopenApproval checkpoint, which forces a human to look at the Sent folder and clear the marker by hand before the send path reopens. Genuine failures, which carry a real error code, stay directly retryable. Only the lie gets a guardrail.

And the verification is the part I insist on, because a security fix you didn't prove both directions is a mood, not a fix (Chapter 3's law, applied to state instead of shell). The regression test drives an approval into sending with a stale timestamp, runs the normalize sweep so it becomes send_failed/send_interrupted, and then attempts a re-approve. On the unpatched code the re-approve succeeds, the test fails with "Missing expected rejection," which is the bug reproduced as a red test, the double-send path provably open. On the patched code the re-approve is rejected, the state holds at send_failed, and the sanctioned reopen-then-send path still works and sends exactly once (sendCount === 1, the both-directions assertion): the lie is blocked and the legitimate recovery still functions. Red before green, full suite green before the PR.

There was a second bug in the same audit, smaller but the same species. rejectRequest, the verb that terminally rejects a pending approval, had no operator gate. Its siblings did: cancelApproval and reopenApproval both call assertOperatorOnly first. rejectRequest checked only that a note was present. Which meant an assistant-tier caller, holding only an assistant bearer token, could reject an approval the operator had already approved and was about to send, unilaterally overriding operator intent through a verb that forgot to ask who was calling. One line closed it: assertOperatorOnly(identity, "reject this approval"), slotted in ahead of the note check, proved red-then-green the same way. Two real holes, two merged fixes, each with a regression test, and not one real email sent during the entire audit; the tests run against a fixture send implementation, because you don't verify a double-send guard by double-sending.

No unified write store, ever

The architecture under all of this deserves its own beat, because it's a deliberate refusal that most people building a "personal dashboard" get wrong.

The control plane is a hub, and it is explicitly not a database of everything. The design is hub-and-spoke: the hub owns the read-and-present layer and the approval-gated action layer, and it owns no source data. Six spokes, bridge-db, the notification hub, the portfolio auditor, the eval suite, the knowledge vault's exporter, the Notion snapshot, each remain the system of record for their own data, and the hub reads from them through declared seams. There's no unified write store, ever. That phrase is load-bearing. The moment you build one database that everything writes into, you have built a single point where every producer's bugs commingle, where a schema migration touches all of them at once, and where "who last wrote this and can I trust it" becomes unanswerable.

Instead, every artifact the hub reads is wrapped with provenance and a freshness verdict. A SpokeArtifact carries when it was generated, when it was read, and a computed freshness derived from the spoke's own declared policy: fresh, aging, stale, or unavailable. And there is a hygiene rule binding it all, which is itself a small instance of the book's thesis: alert-class outputs may derive only from fresh payloads. A stale spoke cannot page you about a risk score, because the score is computed from data you've already flagged as untrustworthy; the only alert a stale spoke is permitted to raise is its own staleness. This is the state machine's "don't lie" rule generalized to the whole hub: a control plane is not allowed to dress up stale data as a live signal, because a human who acts on a confident-looking alert built from week-old inputs is the double-send operator all over again, one layer up.

That rule wasn't free, either. A pre-commit review of the freshness machine caught a defect on its way to shipping: computeFreshness had a NaN leak, an unparseable generatedAt fell through to stale with an age of NaN, which is to say the function would silently mislabel data whose freshness it actually could not determine. The fix routes unknown age to aging with a warning, never silently to stale: degrade loudly, never suppress, and two regression tests pin it. Even the control that enforces honesty had a small dishonesty in it, caught only because someone reviewed the control with the same suspicion it was built to apply.

The class, not the instance

The two bugs were instances. The class closed only after the audit swept the whole service layer for dropped guards and fire-and-forget promises, then proved every flagged site resolved to an awaited call, a Promise.all, or a delegate the caller awaited. The control plane now gates outward action, mints fresh action-bound tokens, refuses a unified write store, wraps inputs in provenance and freshness, and treats the honesty of its own state labels as a security property. A stale spoke cannot page you about a risk score; the only alert a stale spoke may raise is its own staleness. The human should never be asked to act on a confident label built from data the system already knows is old.