[DESIGN] The failure is established — aim at the solution space

This is not a frozen contract. Nothing here is a measurement, a schema, or an interface, and it is expected to be amended — the point is to record what we now know and where it points, not to fix the destination. Amend it freely; the only thing worth preserving is the evidence in the first section.

What is established

SHODANN reviewed its own pull request fourteen times while #61 was open. The result is not a bug list, it is a demonstration:

Every number in every review was correct. Coverage, test counts, commit counts, diagnostics — all exact, every time. What was wrong was the interpretation laid over correct figures: a mechanism invented between two unrelated instruments, functions invented inside a generated markdown file, a promotion path invented for a role that is assigned, a measured zero reported as an unmeasured absence, one quantity reported twice as two different numbers.

Not one of the eleven was visible to the test suite. Not one was visible in a diff. All eleven required reading the posted output.

Why this is the student's problem and not only ours

This is the shape of the thing that happens when a student vibe codes. The code runs. The tests pass. The explanation attached to it is fluent, specific, and confident. What is wrong is not syntax — it is the claims about what the work does, and the tools a student has (does it run? is the suite green?) cannot separate a grounded claim from an ungrounded one. Ours could not either, through eleven classes and ninety-two new tests.

Tightening this, because it matters: what we have demonstrated is that a fluent reviewer produces confident ungrounded claims about work it has not examined, and that green tests do not detect them. That a student fails the same way is a conjecture this system has never tested. Fourteen rounds, one citizen — the author, mid-band, 97.5% coverage, a large suite. The paths the product exists for have never run: a first submission, an all-failing suite, the lowest band, someone with no reference frame. The instrument is proven; the subject was the easiest possible one. Treat the generalisation as the thing to test, not as a premise.

The one law we have

Stated because it is the only finding that generalised past the system it came from, and because it is the whole reason to think there is a teachable move here at all:

A rule enforced by something runnable survived the next run. A rule written as an instruction came back.

Nine classes, thirteen rounds. Every class given a mechanism held. Every class given only prose returned — twice on the very next review, and once from prose shipped in the same commit that caused the defect it forbade. This is not about model quality; the figures were always right. It is about whether the rule had a mechanism behind it.

With a corollary that cost us real money to learn:

A check over a closed vocabulary is cheap forever. A check over an open vocabulary is a sub-department you now staff.

Numbers, headings and word counts are closed — written once, never grown. English phrasings of a claim are open, so the word list grows after every review and every entry is a new false-positive surface. Four of those were opened in one week: 71 hand-maintained terms, and one probe measured rejecting 2 of 9 sentences a correct review would contain. See the deletion list in #61.

The solution space

Options, deliberately not a plan, because the framing above is what needs testing first.

1. Make the student build the detector. The practice that worked here is mechanical and gradeable: write a check, revert the thing it guards, watch it go red, restore. A student who does that has demonstrated the difference between a grounded and an ungrounded claim in a way no prose exercise establishes. The repo already runs this discipline on itself — put the defect back before believing the guard.
2. Grade the guard, not the fix. A submission that adds a passing test proves nothing. A submission that adds a test which fails against the defect it names proves the student located the defect. That is checkable by CI, not by an instructor reading code.
3. Teach the maintenance question at the upper bands. "Is this check worth what it costs to keep?" is engineering judgement, and the closed-versus-open distinction gives it a concrete first answer. This is band-appropriate — it means nothing at INFRARED and it is the whole job by GREEN.
4. Scope by band, as the ladder already does. At the lowest band a student cannot write a probe; they can read output and say what it told them. Reverting one guard is one band up. Writing the check is above that. Deciding whether the check should exist is above that again.

What we are not proposing: grading by absolute skill, detecting plagiarism, or any check a student cannot run themselves. PRD.md §7 already excludes those and this does not reopen them.

What a button-push does first

One step, so this is executable without further negotiation:

Run the review loop against a submission that is not the author's. A first submission, or an all-failing suite, at the lowest band. Everything above rests on a generalisation from one atypical citizen, and one run against a real one is worth more than any further reasoning from this dataset.

Blocked by one thing, and it is a security prerequisite rather than a functional one: the prompt library is located from the working directory, so a second repository supplies the instruction set — in the job holding the write token and the model key. Reproduced. See design_docs/ONBOARDING_A_REPOSITORY.md and #51. Onboarding must not ship before that is fixed.

Where the evidence lives

Start with design_docs/EARLY_RUNS.md, design_docs/ONBOARDING_A_REPOSITORY.md, and src/shodann/groundedness.py, then run the review loop against a first or all-failing submission as described. Compare the result with PRD.md §7 and the band-scoping documents. Done requires evidence about whether the generalisation holds and a decided direction for the solution space.