# The Control Plane

The bug that scared me most did not crash anything. It told the truth about a lie.

In the operator control plane, the local service that gates every outward action my agents take, there
is a state machine that tracks each pending action through its life. It's the thing standing between an
agent's intention to send an email and an email actually leaving the building. An approval is `pending`, then
`approved`, then `sending`, then `sent`. Clean, legible, the kind of state diagram you draw on a whiteboard
and feel good about. And buried in the recovery path was a sequence that could turn one of those labels into a
confident lie.

Here is the sequence, traced through the real code in `app/src/service/review-approval.ts`. When the service
sends an approved draft, `sendApprovedDraft` writes the state `"sending"`, then awaits the actual Gmail call,
then, on success, writes `"sent"`. Three steps. Now kill the process in the gap between step two and step
three: Gmail has *accepted the message*, the email is *genuinely on its way to a real human*, but the
`"sent"` write never lands. The approval is frozen mid-flight in `"sending"`, and no provider message-id is
persisted until the success write, and that is what makes it dangerous: there is no idempotency key, nothing to
dedupe against. Fifteen minutes later a janitor sweep, `recoverStaleSendingApprovals`, finds
the stuck record and relabels it: `state="send_failed"`, `last_error_code="send_interrupted"`. And then the
action gate, `assertApprovalActionAllowed`, cheerfully permits an `approve` from `send_failed`, because of
course it does, a failed send should be retryable.

Read what just happened. The state says **failed.** The reality is **delivery unknown**, the email may very
well have been sent. An operator looking at "failed" does the obvious, correct-seeming thing: re-approve, re-send.
And the recipient gets the same email twice. Real money if it's a billing notice, real reputation if it's a
client. The state machine did not crash. It produced a label, the label was false, and a human trusting the
label took an unsafe action on its authority. That's the whole chapter in one bug: **a control plane's
deepest job is not to act; it is to never lie about what it did, because everything downstream trusts the
label more than the truth.**

## Gate every outward action

Step back to the principle, because the bug only matters in light of it.

An agent that can send email, open pull requests, post to channels, or call third-party APIs has *outward*
reach, it can make things happen in the world that you cannot un-happen. Chapter 3 was about containing what
an agent can do to *your machine*; this chapter is about containing what it can do to *everyone else*. And the
move is the one the whole book keeps making: you don't ask the agent to be careful about sending. You
put a gate between the agent and the outside world, the gate is enforced in code the agent cannot talk past,
and every outward action flows through it.

In the control plane that gate is a four-step flow, and I want to walk the *clean* path, not just the broken one,
because the architecture of the happy path is what makes the broken one fixable. An agent drafts:
`mail_draft_create` writes a draft artifact, nothing leaves. The agent requests approval:
`approval_request_create` mints an `approval_id` and moves the record to `pending`. Then the two steps that an
agent *cannot perform*, approve and send, are operator-only, and each is protected by a confirmation token
whose construction is the point.

The token isn't a password. It's a cryptographic binding. The digest is
`SHA-256(approvalId : action : draftDigest : token)`, which commits, in one hash, to *which approval*, *which
action* (approve is not send), and *which exact draft content*. That binding is what makes replay impossible:
a token minted to approve cannot be replayed to send, a token for one approval cannot be reused on another, and
if the draft text changes after the token is minted, the `draftDigest` no longer matches and the
token is dead. The service enforces that last property directly: mutating a draft nulls the digest and expires
the approval, so there's only ever one outstanding token per approval, bound to exactly the bytes the operator
saw when they decided to trust it. The token has a ten-minute TTL enforced in SQL, not in application code, and
it is consumed by an atomic `UPDATE`-as-compare-and-swap against single-writer SQLite: single-use, no
time-of-check-to-time-of-use window, no way for two sends to both believe they won the token. (The book's
evidence table elsewhere ascribes a specific bit-length to that token; I could not confirm the exact entropy
figure in the session sources for this chapter, so I describe the binding, which is the load-bearing security
property, and leave the bit-count marked `[unverified]` until I read it out of the token-generation code
directly.)

That's the positive path: draft, request, approve-with-bound-token, send-with-bound-token, every irreversible
step requiring a fresh single-use credential the agent can neither mint nor reuse. It's a good design. And a
good design is exactly the kind of thing that lulls you into not auditing it, which is how the double-send bug
lived in the recovery path of an otherwise-solid state machine.

## The state machine's one job

So I audited it, the way Chapters 3 and 5 taught me to audit my own controls: not by reading the code for
elegance, but by reasoning adversarially about where the labels could lie.

The double-send fix is small and it is the right kind of small. The recovery sweep already had a message,
"inspect Sent mail before retrying," but that message was *advisory*. It was a sentence printed for a human,
and nothing in the state machine enforced it. (You will recognize this from Chapter 2: a human instruction is
not a code contract. "Be careful" addressed to a machine is the same non-control as "be careful" addressed to
an agent.) The fix turns the sentence into a transition rule: an approval marked with the `send_interrupted`
error code, which signals delivery-unknown rather than a *real* Gmail rejection (real rejections carry the
provider's actual error code), can no longer be silently re-approved. It must pass through the operator-only
`reopenApproval` checkpoint, which forces a human to look at the Sent folder and clear the marker by hand
before the send path reopens. Genuine failures, which carry a real error code, stay directly retryable. Only
the lie gets a guardrail.

And the verification is the part I insist on, because a security fix you didn't prove both directions is a
mood, not a fix (Chapter 3's law, applied to state instead of shell). The regression test drives an approval
into `sending` with a stale timestamp, runs the normalize sweep so it becomes `send_failed`/`send_interrupted`,
and then attempts a re-approve. On the unpatched code the re-approve *succeeds*, the test fails with
"Missing expected rejection," which is the bug reproduced as a red test, the double-send path provably open.
On the patched code the re-approve is rejected, the state holds at `send_failed`, and the sanctioned
reopen-then-send path still works and sends *exactly once* (`sendCount === 1`, the both-directions assertion):
the lie is blocked and the legitimate recovery still functions. Red before green, full suite green before the
PR.

There was a second bug in the same audit, smaller but the same species. `rejectRequest`, the verb that
terminally rejects a pending approval, had no operator gate. Its siblings did: `cancelApproval` and
`reopenApproval` both call `assertOperatorOnly` first. `rejectRequest` checked only that a note was present.
Which meant an assistant-tier caller, holding only an assistant bearer token, could reject an approval the
*operator had already approved and was about to send*, unilaterally overriding operator intent through a verb
that forgot to ask who was calling. One line closed it: `assertOperatorOnly(identity, "reject this approval")`,
slotted in ahead of the note check, proved red-then-green the same way. Two real holes, two merged fixes, each
with a regression test, and not one real email sent during the entire audit; the tests run against a fixture
send implementation, because you don't verify a double-send guard by double-sending.

## No unified write store, ever

The architecture under all of this deserves its own beat, because it's a deliberate refusal that most people
building a "personal dashboard" get wrong.

The control plane is a hub, and it is explicitly *not* a database of everything. The design is hub-and-spoke: the
hub owns the read-and-present layer and the approval-gated action layer, and it owns *no* source data. Six
spokes, bridge-db, the notification hub, the portfolio auditor, the eval suite, the knowledge vault's
exporter, the Notion snapshot, each remain the system of record for their own data, and the hub reads from
them through declared seams. There's no unified write store, ever. That phrase is load-bearing. The moment you
build one database that everything writes into, you have built a single point where every producer's bugs
commingle, where a schema migration touches all of them at once, and where "who last wrote this and can I
trust it" becomes unanswerable.

Instead, every artifact the hub reads is wrapped with provenance and a freshness verdict. A `SpokeArtifact`
carries when it was generated, when it was read, and a computed `freshness` derived from the spoke's own
declared policy: `fresh`, `aging`, `stale`, or `unavailable`. And there is a hygiene rule binding it all, which
is itself a small instance of the book's thesis: **alert-class outputs may derive only from `fresh` payloads.**
A stale spoke cannot page you about a risk score, because the score is computed from data you've already
flagged as untrustworthy; the only alert a stale spoke is permitted to raise is its *own staleness.* This is
the state machine's "don't lie" rule generalized to the whole hub: a control plane is not allowed to dress up
stale data as a live signal, because a human who acts on a confident-looking alert built from week-old inputs
is the double-send operator all over again, one layer up.

That rule wasn't free, either. A pre-commit review of the freshness machine caught a defect on its way to
shipping: `computeFreshness` had a NaN leak, an unparseable `generatedAt` fell through to `stale` with an age
of `NaN`, which is to say the function would silently mislabel data whose freshness it actually could not
determine. The fix routes unknown age to `aging` with a warning, never silently to `stale`: *degrade loudly,
never suppress,* and two regression tests pin it. Even the control that enforces honesty had a small dishonesty
in it, caught only because someone reviewed the control with the same suspicion it was built to apply.

## The class, not the instance

The two bugs were instances. The class closed only after the audit swept the whole service layer for dropped
guards and fire-and-forget promises, then proved every flagged site resolved to an awaited call, a `Promise.all`,
or a delegate the caller awaited. The control plane now gates outward action, mints fresh action-bound tokens,
refuses a unified write store, wraps inputs in provenance and freshness, and treats the honesty of its own state
labels as a security property. A stale spoke cannot page you about a risk score; the only alert a stale spoke may
raise is its own staleness. The human should never be asked to act on a confident label built from data the system
already knows is old.