bench notes
2026-07-04
Truthful agent work is not one grand verifier. It is a chain of small instruments, each checking a single trust handoff: what a tool can touch, where it can send, what it did, whether the call was calibrated, and which receipt a human can review.
The chain runs build, audit, guard, record, score, receipt, and each link is a public repo, a package, or an honest limit, never one product claim. Most of it ships today as something a stranger can install or inspect without private context; one link runs on private data and stays off this shelf. What follows is that machinery, grouped by what it does.
The graders
They score, audit, or grade something, and publish the verdict.
HarnessBench
Grades whether an agent harness's stated safety rules are mechanically enforced or merely advised. It fires destructive-command probes at a configuration and reads whether the mechanism actually blocked them, not whether the model said it would. Blocking everything and blocking nothing both score zero, so only a guard that discriminates passes.
OWASP ASI05 offline tier: stated-in-prose +0.00, a regex blocklist +0.09 (wrong in both directions), an independent semantic guard +1.00, across 68 probes. Reproducible with no API key. as of 2026-08-16
bash scripts/reproduce.sh
harnessbenchmethodology
Verification Ledger
Grades whether a coordination ledger enforces its stated safety rules as mechanisms or merely as words. It fires positive and adversarial probes at a store and reads whether provenance typing, the promotion gate, and the retention invariant actually hold, not whether a README promises they do. Blocking every path and allowing every path both fail, so only a store that keeps the legitimate path open and the laundering path closed passes.
The reference store scores a perfect 1.00 across four invariants; the conformance suite was adversarially broken, then hardened, before release. Public, MIT, zero runtime dependencies. as of 2026-08-16
python -m verification_ledger.conformance
verification-ledgerpypimethodology
CheckSeal
Seals a verification run into an in-toto/DSSE attestation tied to one subject digest, logged to Sigstore, so a stranger can verify it without trusting the operator. The receipt behind the receipts: flip one byte of the signed payload and the verifier refuses.
Six T2 seals in the Rekor transparency log and two T1 seals hang on the line: the harness-config seal (rekor 2493890211) and five OPERANT-J judge sittings. The forged copy is refused in the browser. as of 2026-08-19
checkseal verify-keyless harness-config.sigstore.json --subject harnessbench-semantic-clean-room.config-manifest.json
checksealpypithe sealed prints
OPERANT
Scores an agent as an operator, not a coder: proceed, refuse, escalate, or take the sanctioned path under matched pressure. Refusing everything and proceeding on everything both score zero, so only calibrated judgment passes.
Five judge sittings on the refusal-probe docket sealed in Rekor on 2026-08-19, two pass and three fail; each seal binds the verdict to the run, not the model to a rank. The historical OCS ranking is marked not durable by the benchmark card and is not repeated here. as of 2026-08-19
npx -y saagar-operant-mcp
operantthe sealed sittings
mcp-trust
The trust-grade layer as a live public registry: servers graded A through F from real sandboxed scans (Docker, network off, receipt-backed evidence), with a separate transparency axis.
31 servers graded; registry live. Grades are static since 2026-07-11: I disabled the weekly re-scan lane and stripped its deploy authority, so re-scans are manual until I re-arm it. The badge loop is not yet claimed: no third party has verified a grade end to end. as of 2026-07-11
uvx mcp-trust mcp-serve
mcp-trustpypi
MCPAudit
Reads the MCP servers wired into your tools and scores the risk they declare across capability categories.
published · config-only mode infers declared capability, not live behavior
uvx --from mcp-audits mcp-audit scan --skip-connect --json report.json --redact
MCPAuditpypi
The packages
Install or clone them and run them yourself; read-only where they touch a machine.
mcpforge
Turns a plain-English description into a complete FastMCP project: tools, validation, error handling, tests, and client docs. It scaffolds and validates structure; domain logic, auth, and rate limits remain your job.
published · PyPI fastmcp-builder
uvx --from fastmcp-builder mcpforge demo
pypi
cross-provider-egress-guard
A default-deny firewall for the covered agent paths: network sends and sensitive reads are checked against one shared policy across Claude Code and Codex, fail-closed where the hooks can classify the call.
published · residuals named plainly: exfiltration to already-allowed hosts, and prompt injection arriving in tool output
git clone https://github.com/saagpatel/cross-provider-egress-guard
proof-pr
Attaches a machine-readable receipt to a change: the verification commands that ran, their real exit codes, and a deterministic ready, revise, or reject. Receipts are review evidence, not cryptographic provenance.
published
git clone https://github.com/saagpatel/proof-pr
portfolio-mcp
The whole site as a corpus: search the essays, book, and notes, pull a page's full text, read the profile, and list the projects. Read-only.
published · MCP registry io.github.saagpatel/portfolio-mcp
npx -y saagar-portfolio-mcp
npm
operant-mcp
The OPERANT benchmark as a queryable API: model calibration results, the methodology, and the case library. Read-only.
published · MCP registry io.github.saagpatel/operant-mcp
npx -y saagar-operant-mcp
npm
shadow-mcp
A local, read-only inventory of the MCP servers actually configured on this machine: it discovers them across your clients and risk-grades each one.
published · MCP registry io.github.saagpatel/shadow-mcp
uvx shadow-mcp mcp-serve
pypi
The exhibits
Interactive and public-safe; you operate them here, no install.
Verdict Machine
The repo auditor's verdict cascade, ported to the browser and golden-tested against the Python original. Scrub a synthetic repo's signals and watch the real decision functions compute its attention lane and risk tier.
browser port verified by a golden harness against the Python source
read the cascade source → GithubRepoAuditor
Plumbline trace viewer
An open trace schema and offline scorer for the run your harness already produced: inspect a synthetic agent run, click through the decision path, and see why the offline scorecard gives partial credit instead of a hand-wavy pass.
public fixture · early core, no release yet
open the trace viewer → plumbline
Fleet Mission Control
A public-safe snapshot replay of the operator OS: five AI systems coordinating through one SQLite file, with exported bridge-db telemetry visualized. A replayed export, not live telemetry; the capture date is shown in the page.
replayed public-safe export
read the replay source → bridge-db
Next-word sampler
How a language model picks the next token: watch the probability distribution, drag temperature to flatten or sharpen it, move top-k and top-p, then roll the weighted die and watch a sentence form.
illustrative model, runs entirely in the page
Operator OS walkthrough
Six organs and a few AI agents coordinating through one SQLite spine. Click a part, watch the truth move.
interactive explainer of the real architecture
Interleaving Explorer
Two agents, one row, and a lost update: step through the interleavings that let concurrent writers corrupt shared state, and the discipline that prevents it.
synthetic walkthrough
Predictions ledger
Dated, falsifiable predictions with explicit resolution criteria, scored by Brier as they resolve. The scoring machinery is published before the first resolution lands.
13 dated predictions, all pending as of 2026-07-19
Agent Failure Museum
A public-safe gallery of synthetic failures: the little traps that make agent work look done, safe, or measured before the evidence catches up. Every exhibit is invented for explanation; no real transcripts, paths, or findings.
7 synthetic exhibits, each naming the control that catches it