monad-developers / monad-developers/ultrafuzz
Claims in comments, docs and PR bodies routinely outrun what executes; five instances in five review rounds
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 85
- Forks
- 18
- Avg merge
- 11h 10m
- Merged PRs (30d)
- 194
Description
Problem
The same defect has now been caught in five consecutive review rounds: a sentence that claims more than what executes, or more than what was actually verified. Each instance passed its author's own review and was caught only by independent verification.
- A configuration surface that parsed, validated and reported bindings no adapter ever read. Closed as superseded.
- Documentation asserting that provider and harness "are not welded together", false for three of the four shipped agent refs — one hardcodes its endpoint, one exposes no base-URL option at all, one is welded to its vendor.
- A code comment asserting a list was "every root resolves state from". Falsified once, then the list was quietly extended twice more.
- A pull-request body headlining "a real run" for what was an argv spawn: no node, no workflow, no run record.
- A compliance table asserting "every throw names the config path and the variable", falsified by two of the four throws in the same change.
The common shape is an unqualified universal — "every", "all", "no run writes", "never appears" — asserted where only a subset was established. These read as stronger review evidence than a scoped claim, which is exactly why they survive review.
Existing guards do not cover this. Tests assert behaviour, not prose. Lint and formatters do not read English. Review catches it only when a reviewer independently re-derives the claim, which is expensive and inconsistent.
Proposal
- A claim in a comment, doc, or PR body is subject to the same evidence bar as a test assertion. If a sentence would fail as an assertion, it must be scoped until it would pass.
- Add a review-checklist item: for each universal in a change, name the evidence that establishes it, or rewrite it as the subset that was checked. Prefer the scoped sentence over the impressive one.
- Treat a docs-only change that introduces a forward-looking architectural claim as out of scope for that change. Two of the five instances above were docs edits whose stated purpose was factual accuracy.
- Where a claim is genuinely load-bearing, back it with a discriminating test — one that fails when the property stops holding. A test that cannot fail reads as coverage while providing none; one such assertion was found and replaced during this work.
- When a claim is scoped rather than proved, say what was checked and against which version, so a later reader can tell verification from assertion.
Non-goals
- Weakening claims that are true and established.
- A linter for English. The gate is reviewer attention, directed by a checklist.
Acceptance criteria
- The review checklist names universals as a category requiring evidence or scoping.
- Docs-accuracy changes are expected not to introduce new architectural claims.
- A load-bearing property is expected to carry a test that fails when the property does.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file, checklist location, or test path is named. Start by locating the repository's review checklist and docs-accuracy guidance, then check the three acceptance criteria; done means universals require evidence or scoping, docs-only accuracy changes avoid new architectural claims, and load-bearing properties have discriminating tests.
Written by the indexing model from the issue text.
Assessment
- Domain
- developer-experience, documentation, testing-qa
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100