docs-review: the update lane and the no-claims composer stub can both publish an unvalidated review body
@CamSoper is already working on this.
Since Jul 30, 2026.
- Dominant language
- HTML
- Stars
- 178
- Forks
- 271
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 557
Description
Two gaps in the docs-review pipeline, both surfaced by the 2026-07-30 fork regression battery that gated #20546 / #20518 / #20577. Neither blocked those PRs; both let a structurally broken review reach a reader without anything flagging it. They're filed together because they share a root shape: a review body can lose required content and nothing checks.
1. The re-entrant update path never runs the validator
claude-update.yml runs exactly one validation step:
python3 .claude/commands/docs-review/scripts/validate-pinned.py \
count-buckets --pr "$PR" --repo "$REPO" > /tmp/bucket-counts.txt
count-buckets counts bullets. It does not check structure. The full
validate → splice → validator-fix chain that claude-code-review.yml runs after its
editorial pass never executes here, so whatever the model writes on a #update-review
is published as-is.
Observed on a fork fixture: after the first @claude #update-review, the pinned body had
lost ### 🔍 Verification trail, ### 📊 Editorial balance, and the investigation-log
block. Running the real validator against that published body reports 4 violations:
investigation-log-block-presentmandatory-h3-ordercount-table-matches-bulletsbucket-bullet-line-range-prefix
The loss is permanent: update.md has the re-entrant path merge from the prior pinned
comment, so once the evidence spine is gone, every subsequent update inherits a body that
no longer has it. A PR that gets one fix push and then merges carries a review with no
verification trail at all.
The immediate trigger was --effort low on that lane (2/2 runs dropped the sections; the
no-effort control kept them), which is why #20518 shipped without it. But the effort flag
is the trigger, not the cause — the cause is that this lane has no structural gate, so any
model, any effort, any future prompt edit can do the same thing silently.
Ask: run the same validate → splice → validator-fix chain on the update path before
upsert, or at minimum fail loudly when a published body drops a mandatory H3.
Related: this is the gap listed under Follow-ups in #20518 as
"candidate-claims-coverage is unenforced on the re-entrant update path" — the observed
loss is broader than that one rule.
2. The not run (no claims in this diff) stub strands the investigation-log bullet
When .verified-claims.json has no fact-check verdicts, compose-review.py writes:
- **External claim verification:** not run (no claims in this diff)
That form is explicitly allowed by check_external_claim_state_format, so it validates
fine — but it carries no · N specialists (…); K cross-specialist corroborations and
no · routed: I inline, P Pass 1, F Pass 2 segments, because the composer has no counts
to put there.
If the reviewer then finds claims in-review (which is legitimate and expected), it rewrites
that bullet into the X of Y claims verified (…) form — and now the two metadata rules
apply to a line that was never given a skeleton to preserve. Both fire, validator-fix
can't reliably reconstruct the counts, and the review publishes with a CI annotation.
Observed once on fork fixture #222 (mirroring #18799): composer stubbed not run, the
reviewer added 3 in-review claims, published with external-claim-dispatch-metadata +
external-claim-routed-metadata. The same fixture converged to 0 violations on a re-run,
so it's discovery-dependent rather than deterministic — which is exactly why it'll keep
recurring at a low rate.
Ask: have the composer emit the metadata skeleton (zeros are fine, and honest) even on
the not run path, so a later in-review addition has something to preserve and increment.
Evidence
Full battery write-up, per-run captures, and the A/B receipts:
scratch/2026-07-30-triple-regression/REGRESSION-SUMMARY.md (§5.B and §5.C).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.