# How a Grader Earns the Right to Grade

I wrote a tool that reads your agent harness and hands back a letter grade. A to F, the whole insulting alphabet. An F is not a diagnostic; it is an accusation. So the fair question, the one I had to answer before I could point the thing at anyone else's setup, is what gives a stdlib script the standing to make that accusation at all.

The answer is not the rubric. Every grader has a rubric. Authority comes from what the grader is willing to do to itself.

## The rubric is the easy part

harness-scorecard scores ten rubric dimensions into a single weighted grade, and I want to be honest that the weighting is the least interesting thing in the tool. Weights are a matter of taste, and a grade assembled purely by adding up weighted points has a fatal property: you can climb it by accumulating cheap wins while the one load-bearing thing stays broken. A harness can pass nine easy checks, fail the tenth that actually matters, and post a respectable B. That is a score measuring effort, not safety, and a score that measures effort will always be gamed by whoever needs the number to look good.

So the first thing that earns a grader its authority is a gate that the points cannot buy their way past. harness-scorecard has capability gates: a critical hole caps the grade no matter how much easy credit sits on top of it. You cannot score an A with readable credentials. It does not matter how many cheap checks pass; a harness with world-readable secrets is not an A harness, and a grader that would call it one is measuring the wrong thing. The gate is the grader refusing to let its own arithmetic lie on its behalf.

## Refuse to credit what you cannot prove

The harder problem is that a grader is itself a piece of software that can be fooled, and the most tempting way to fool it is to let it be generous.

Here is the specific trap. Many harnesses route their safety checks through an opaque hook dispatcher, so the grader cannot see the individual checks directly; it sees one dispatcher and a pile of indirection. The cheap move, the move that inflates every score, is to credit a check because the dispatcher's comments or a config key mention it by name. Read the prose, believe the prose, hand out the points.

harness-scorecard does not do that, and refusing to is most of what earns it the right to grade anyone. It reads the dispatcher's actual source and credits a check only against a code anchor: a regex, an identifier, a call, a path-literal. Never a prose match. And it suggests coverage rather than auto-crediting it, because a suggestion invites you to check and a credit invites you to relax. Then it goes one step further, the step I would look for in any grader before I trusted its letter. Every pattern it matches is paired with an oracle test that pins the false-credit boundary: a check that would award points to sanitize-path-style prose, or to a bare file extension, is defined as a bug in the grader, not as evidence about the harness. The tool ships tests whose entire job is to enumerate the ways it is not allowed to be fooled.

That is the tell. A grader with authority can name its own failure modes out loud and has already written a test for each one. A grader without authority asks you to trust that it read carefully.

## Draw your own blood first

The last thing, and the one people skip, is that the grader has to grade its author before it grades a stranger.

harness-scorecard's first serious target was not a demo harness built to show off a clean A. It was my own Codex configuration, the real one, running on the machine I work on every day. The grade came back low, and it was right that it came back low, and that grade became the work-list for a measured hardening pass that moved the harness from a D to a B. The tool's first casualty was its own author's infrastructure, which is exactly the order a tool like this should draw blood in. If the first thing your grader does is find fault with everyone but you, it is not a measurement instrument. It is a sales pitch with a rubric attached, and the rubric is there to make the pitch sound like a finding.

Grading yourself first does two things at once. It proves the grader bites, on a target that could not have been rigged, because you do not rig the exam you are about to fail in public. And it forces the grader to be survivable to its own author, which is the only user guaranteed to know when the grade is wrong.

There is a fourth leg the tool has not finished earning yet: validity. A grader that caps its own score, anchors credits in code, and draws blood on its author has earned standing to be listened to. It has not, from those facts alone, proved that its letters predict independently discovered failures. That proof has to accrue the boring way: grades compared against defects found later by other routes, misses filed as misses, false confidence treated as a bug in the grader. Until that ledger exists at scale, the letter is a disciplined opinion with receipts, not a final verdict.

## The standard is the one you already apply

Put the four together and they are a single posture wearing four coats. Cap your own score so your arithmetic cannot flatter a broken harness. Credit only what you can anchor in code, and keep a test for every way you might be fooled into crediting more. Grade yourself first, on real infrastructure, and let the low grade stand. Then keep asking whether the grade predicts failures the grader did not already know about.

None of that is about being clever. It is about a grader holding itself to the exact standard it holds everything else to, which is the standard this whole series keeps circling: an account of your own coverage is not evidence, especially when the account is coming from the thing doing the checking. A grader that exempts itself from that has not earned the right to grade your harness. It has earned the right to have an opinion about it, printed in a serious-looking font, which is a different and much smaller thing.

The letter on the scorecard is not where the authority lives. The authority is in whether the grader was willing to earn the letter it gave itself.