The Field Manual

The Lying Instrument

Part I: The World, Misreported

You are here if

A number is asking you to act on it, and something about the number is off. The month's spend came in at a quarter of the invoice. The orphan scan found dozens of dead files in a vault you tend weekly. The eval says the cheap model beats the premium one, decisively. The gate scanned your change and found zero pages to check. The reading is clean, specific, machine-produced, and you do not believe it.

You are here if the thing you distrust is not the system under test but the tool you pointed at it: the parser that scores the answers, the meter that totals the sessions, the scanner that counts the orphans, the wrapper that times the suite. You are also here, most urgently, if nothing is surprising yet: if you are about to spend an unrepeatable run on an instrument you have never fed a known answer.

You are in the wrong chapter if the signal is the system's own self-report: that is chapter 1. If the reading was true when it was made and the world has moved since, that is chapter 2. If a worker you dispatched wrote the report, that is chapter 6. This chapter is for the second thing, the tool that stands between you and the world and hands you the world as a number.

The mechanism

An instrument, here, is anything that turns the world into a reading: an answer parser, a cost meter, a repo inventory, an eval harness, and the rig around them: the shell timeout wrapping the suite, the reset mechanism poking the app, the order two streams were concatenated in. None of it is the world; all of it is authored software, and it fails the way authored software fails. What makes the class its own chapter is the shape of the failure: silent, and formed like a valid answer. A parser that cannot read an answer does not error; it scores the answer absent. A pricing table missing a model does not complain; it prices the tokens at zero. A broken instrument and a working one return the same type. The lie is well-formed.

The type specimen. An evaluation capture that could not be re-run was scored by a frozen answer parser. The parser's regex expected the answer bare; the model returned it wrapped in standard markdown bold, and the parser, unable to see through the asterisks, scored a correct answer as missing. It did not warn. It emitted a lower score with the confidence of any other score. The parser even had a self-test, and it passed, because it fed the parser clean, unformatted input: written by the same hand as the regex, it inherited the same assumption. The gap surfaced only when the surprising score was walked back to the raw transcript, where the valid answer sat, in bold. Both halves of this chapter's discipline are in that incident: the variant-feeding that would have caught the regex before the capture, and the walk back that caught it after.

Two things make the class dangerous. First, instruments are the least-tested software you run. The system under test gets your suspicion; the ruler you hold against it gets none. Second, timing: you reach for an instrument at your most disciplined moment, mid-verification, and the reading arrives wearing the credibility of the check you were performing. Distrust is a budget, and you had already spent it one level down. Book One tells the full story of a measurement that lied, and lands the rule this chapter inherits as a field procedure: the apparatus is part of the system under test.

The discipline

Stress-test the instrument before the run that matters, and re-check any surprising number against raw evidence before acting on it.

The ritual:

  1. Feed it a known answer first. Run the instrument on input whose correct reading you already know, including one realistically messy variant: formatted output, out-of-order events, the second schema. Pass: it reproduces the known value on the clean case and the messy one, and you wrote the expected value down before running.
  1. Break it on purpose and watch the failure. Feed it an input it cannot handle: an empty stream, a malformed record, a model it has no price for. Pass: it fails loudly, with an error or an explicit unknown, never with a plausible number. An instrument that answers zero when it means I don't know has just told you what its every future zero is worth.
  1. Enumerate what it reads. List the sources that exist, the subagent sidechains, the secondary indexes, the linked worktrees, against what the instrument actually consumes. Pass: the source list came from listing the live world, not the instrument's documentation.
  1. Name every layer it runs through, and send the known answer through each. Between the world and the reading sits more than the scorer: the wrapper's timeout, the reset mechanism, the concatenation order, all of it instrument. Name each layer, then route the known answer from step 1 through every one and confirm none silently drops it. Pass: every layer is named from the live pipeline, not the instrument's diagram, and the known answer survives each layer in turn.
  1. Interrogate every surprising reading before acting on it. Zero is surprising. Exactly round is surprising. A factor of four is surprising. Decisive is surprising. Trace one case by hand from raw evidence to reading: transcript to score, invoice to total, file to orphan flag. Pass: the hand-traced case matches the instrument, or the number is quarantined and the instrument is now the incident.
  1. Confirm the aggregate through a second channel. Derive the number a different way: the platform's other endpoint, the total re-computed from raw records, the corpus-level property behind the per-case passes. Pass: two independently derived readings agree, and any disagreement is treated as the finding, not as noise.

The case ledger

Eighteen entries, grouped by the kind of instrument that lied: the scorer, the meter, the scanner, and the rig around them all.

The scorer. The grading layer, wrong about the work.

[E089] The author's own tests share the author's blind spots (2026-05-31 / 2026-06-19) Receipt: an author-written test suite passed cleanly on a new analyzer module, but an adversarial review pass caught a real bug the author's fixtures never exercised. Rule: review every new analyzer, detector, or parser module before committing, expect real findings, and verify each with a live check rather than trusting the review outright.

[E098] Per-case gates miss corpus-level exploits (2026-07-04) Receipt: an evaluation corpus passed every individual case, yet a cheap uniform policy could still win across the whole corpus. Rule: after authoring any eval corpus, check its aggregate properties separately; a perfect per-case pass rate does not prove a sound corpus.

[E099] Mixed schemas and brittle vocabulary produce phantom scores (2026-06-12 / 2026-06-19) Receipt: mixed record schemas in an eval index produced phantom zero-recall readings, and a keyword scorer's brittle vocabulary nearly produced a false regression claim. Rule: re-check any eval-harness number against the actual transcripts; report scores as scored, and attach transcript evidence to any surprising number.

[E100] The headline comparison was a broken batch (2026-05-12 / 2026-06-28) Receipt: a headline one-agent-type-dominates result and a single low-recall data point both dissolved, once traced to raw evidence, into a broken batch and an n-of-1 sample. Rule: never tune from a single eval data point; treat harness numbers as directional, and confirm a baseline with a real fixture suite before mutating anything.

[E101] The frozen parser drops valid answers over bold formatting (2026-07-06) Receipt: an unrepeatable evaluation capture silently lost a valid answer because the frozen parser's regex did not tolerate standard markdown bold, and a clean-input self-test hid the gap entirely. Rule: stress-test the answer parser against realistic formatting variants before any unrepeatable capture; if the scorer is frozen, normalize the input in your own harness.

[E102] The eval assumed a plan and got an execution (undated) Receipt: given tools and an imperative-sounding prompt, an agent executed a small task instead of planning it, corrupting scores built on the assumption it would plan. Rule: instruct plan-elicitation agents to decide staffing without performing the work, and hand-check one case for planning, not executing, before trusting the scores.

The meter. The counting layer, short and silent about it.

[E017] Two installs of one cost tool, two schemas (undated) Receipt: two installs of the same cost-tracking CLI returned different JSON schemas, and a sync script that handled only one of them silently dropped every session. Rule: detect a tool's output schema by which key is present and handle every known format; never assume a single response shape.

[E074] One wrong field zeroes all delegation accounting (2026-06-20) Receipt: an assumption about which transcript field marks a subagent turn was structurally wrong, silently zeroing delegated-work accounting across every session it touched. Rule: verify the live field name against a fresh transcript before building instrumentation on it, and dry-run across varied sessions before trusting a derived metric.

[E086] Session self-totals exclude the subagents (undated) Receipt: a session's self-reported totals covered only its main conversation stream, silently excluding everything the subagents did. Rule: aggregate the main stream and every subagent sidechain before treating any self-reported number as the session total.

[E136] Offline pricing silently zeroes a premium model (2026-06-19) Receipt: an offline cost-calculation mode priced a premium model's tokens at zero, understating a month's actual spend by roughly a factor of four. Rule: never use the offline pricing mode for costs that include newer premium model variants; let the tool fetch live pricing.

The scanner. The inventory layer, missing part of the world or believing a fake.

[E044] The gate that scanned zero pages (2026-06-28) Receipt: a site's verification gate silently scanned zero pages when run from a nested worktree, because that path was deliberately excluded to avoid double-counting. Rule: for isolated, gate-requiring work on that project, create a sibling worktree outside the harness's own worktree directory, not a nested one, so the gate can see it.

[E052] The inventory blind to worktrees and nested repos (2026-06-26) Receipt: a repo inventory keyed on a plain directory check missed linked worktrees and nested repos, nearly flagging in-progress branches and unbacked commits as deletable clutter. Rule: detect git-backed directories with a check that handles worktree gitlinks, walk a few levels deep for nested repos, and re-fetch before judging any branch merged or unique.

[E093] One empty metadata field, one false premise (2026-06-19) Receipt: a repo-listing command's license field came back empty for a repo with a valid, recognized license, nearly driving a mass add-missing-license mutation on a false reading. Rule: never conclude a repo has no license from one metadata field; cross-check the platform's separate detection endpoint before acting on a batch.

[E104] Auto-recovery filler fakes a quality metric (undated) Receipt: an automated recovery process wrote generic filler into a repo's context block, flipping a quality metric to read as real on fabricated content. Rule: when clearing a low-quality flag, treat any managed block already present as likely filler and replace it with real harvested content rather than trusting it.

[E151] The orphan check blind to the secondary index (2026-06-19 / 2026-07-03) Receipt: a memory-hygiene check's orphan detector scanned only the primary index, so properly archived files listed in a secondary index were all misreported as orphans. Rule: when a hygiene check reports an unusually large orphan count, cross-reference every secondary index before treating the orphans as real.

The rig. The tool was sound; the apparatus around it was the lie.

[E030] The reset rig manufactures the failure it checks for (2026-06-06) Receipt: screen-capture verification of a desktop app worked cleanly, but a keystroke-injection reset for the same check leaked into an unrelated window and produced a false content-lost reading. Rule: use screen capture and image review freely for visual verification, but hand nontrivial keyboard or mouse automation back to the operator to avoid cross-window side effects.

[E107] The short timeout that impersonated a leak (2026-05-16) Receipt: a short shell timeout around a long test suite produced a mid-test kill that read exactly like a real resource leak, and the fictitious defect got chased. Rule: drive test pacing with the runner's own per-test timeout and let a legitimately long suite finish; suspect a real leak only if the signal appears after all tests report complete.

[E154] First and last are not min and max (undated) Receipt: a parser computing a time span over a concatenated multi-source stream took the first and last events in array order as its bounds; the sources were not chronological, and a multi-hour span collapsed to seconds. Rule: compute merged-stream bounds with explicit min and max on a parsed comparable value, and test with deliberately out-of-order fixtures.

Adjacent doors

If the reading is the system's own report of its own success, an exit code, a green check, a merged banner, that is chapter 1. The dividing question is who produced the reading: the operation reporting on itself is chapter 1; a second thing you pointed at it is this chapter. The two trade readers on purpose: chapter 1 ends by telling you to read the bytes, and the moment you suspect the tool doing the reading, you are back here.

If the number came from a worker you dispatched, a completion summary, a self-scored result, a delegate's own account of its diff, that is chapter 6. An instrument is wrong by accident and wrong the same way every time; a delegate's report is shaped toward your approval, and the discipline there is reading the diff, not calibrating the reporter.

If the instrument that fired is one of your own guards, a gate that just blocked you and looks wrong, keep your hands off its threshold and go to chapter 8. This chapter's instruments inform you, and when one lies, you recalibrate it. A guard that misfires is a different object: narrow it in the heat of the block and the narrowing outlives the incident. The tripwire you are arguing with is not a ruler, and it has its own chapter.