A comment-only change to a checker cannot satisfy check-pr-size's mutation-evidence contract
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: -
Owed by .agents/specs/bench-rocm-llamacpp-denominator.md under ## Owed.
scripts/check-pr-size.py requires every change to a governance_checker path
to carry semantic mutation evidence in its paired suite: the suite must fail
against the BASE checker and pass against the HEAD one. That is the right rule
for a semantic change. It also fires on a change that has no semantics at all,
so a stale COMMENT inside a checker cannot be corrected.
Reproduced, not argued -- a one-line comment edit to
scripts/check-agent-record.py, with no paired test change:
$ python3 -c "... change_errors([ChangedPath('scripts/check-agent-record.py', 1, 1)])"
["checker change 'scripts/check-agent-record.py' requires semantic mutation
evidence in tests/scripts/test_agent_record.py"]
There is no honest way to satisfy it. A comment produces no behaviour
difference, so any test written to pair with it passes against the base checker
too, and the contract then reports BASE checker stayed green ... not semantic evidence. The only routes out are to fabricate a mutation, or to leave the
comment wrong. Both are worse than the trap.
The live instance. scripts/check-agent-record.py:463 says
BACKEND-GATE-ROCM-LLAMACPP is "INVENTORIED, no owner, no spec of its own".
The row now has .agents/specs/bench-rocm-llamacpp-denominator.md, so the
comment is stale, and #2497's repair pass declined to fix it for exactly this
reason rather than inventing evidence for a comment.
Shape of a repair, not a decision. The class the contract is defending is a
change to what the checker DOES. A change that leaves the module's behaviour
identical is not in that class, and the checker can tell the difference without
a network call or a judgement: compare the two revisions with comments and
docstrings stripped, or with the parsed AST, and exempt only when they are
equal. That keeps every semantic change under the contract and stops the gate
from locking prose. Whoever takes this should confirm the exemption cannot be
reached by a change that alters a string literal a message is built from --
AGENTS.md says a checker's message defines what it enforces, so a message is
behaviour and must stay inside the contract.
Found while repairing the fresh review of #2497.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with scripts/check-pr-size.py and its tests, then inspect scripts/check-agent-record.py:463 and tests/scripts/test_agent_record.py using the reproduction in the issue. Compare the proposed exemption against AGENTS.md, especially message changes; done means comment-only checker edits no longer require mutation evidence while behavioral changes still do.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100