pulumi / pulumi/docs

docs-review: the update lane and the no-claims composer stub can both publish an unvalidated review body

Open
#20,596 0 comments 0 reactions 1 assignee View on GitHub

@CamSoper is already working on this.

Since Jul 30, 2026.

area/automation area/docs-content domain:infra
Dominant language
HTML
Stars
178
Forks
271
Avg merge
1d 2h
Merged PRs (30d)
557

Description

Two gaps in the docs-review pipeline, both surfaced by the 2026-07-30 fork regression battery that gated #20546 / #20518 / #20577. Neither blocked those PRs; both let a structurally broken review reach a reader without anything flagging it. They're filed together because they share a root shape: a review body can lose required content and nothing checks.

1. The re-entrant update path never runs the validator

claude-update.yml runs exactly one validation step:

python3 .claude/commands/docs-review/scripts/validate-pinned.py \
  count-buckets --pr "$PR" --repo "$REPO" > /tmp/bucket-counts.txt

count-buckets counts bullets. It does not check structure. The full
validate → splice → validator-fix chain that claude-code-review.yml runs after its
editorial pass never executes here, so whatever the model writes on a #update-review
is published as-is.

Observed on a fork fixture: after the first @claude #update-review, the pinned body had
lost ### 🔍 Verification trail, ### 📊 Editorial balance, and the investigation-log
block. Running the real validator against that published body reports 4 violations:

  • investigation-log-block-present
  • mandatory-h3-order
  • count-table-matches-bullets
  • bucket-bullet-line-range-prefix

The loss is permanent: update.md has the re-entrant path merge from the prior pinned
comment, so once the evidence spine is gone, every subsequent update inherits a body that
no longer has it. A PR that gets one fix push and then merges carries a review with no
verification trail at all.

The immediate trigger was --effort low on that lane (2/2 runs dropped the sections; the
no-effort control kept them), which is why #20518 shipped without it. But the effort flag
is the trigger, not the cause — the cause is that this lane has no structural gate, so any
model, any effort, any future prompt edit can do the same thing silently.

Ask: run the same validate → splice → validator-fix chain on the update path before
upsert, or at minimum fail loudly when a published body drops a mandatory H3.

Related: this is the gap listed under Follow-ups in #20518 as
"candidate-claims-coverage is unenforced on the re-entrant update path" — the observed
loss is broader than that one rule.

2. The not run (no claims in this diff) stub strands the investigation-log bullet

When .verified-claims.json has no fact-check verdicts, compose-review.py writes:

- **External claim verification:** not run (no claims in this diff)

That form is explicitly allowed by check_external_claim_state_format, so it validates
fine — but it carries no · N specialists (…); K cross-specialist corroborations and
no · routed: I inline, P Pass 1, F Pass 2 segments, because the composer has no counts
to put there.

If the reviewer then finds claims in-review (which is legitimate and expected), it rewrites
that bullet into the X of Y claims verified (…) form — and now the two metadata rules
apply to a line that was never given a skeleton to preserve. Both fire, validator-fix
can't reliably reconstruct the counts, and the review publishes with a CI annotation.

Observed once on fork fixture #222 (mirroring #18799): composer stubbed not run, the
reviewer added 3 in-review claims, published with external-claim-dispatch-metadata +
external-claim-routed-metadata. The same fixture converged to 0 violations on a re-run,
so it's discovery-dependent rather than deterministic — which is exactly why it'll keep
recurring at a low rate.

Ask: have the composer emit the metadata skeleton (zeros are fine, and honest) even on
the not run path, so a later in-review addition has something to preserve and increment.

Evidence

Full battery write-up, per-run captures, and the A/B receipts:
scratch/2026-07-30-triple-regression/REGRESSION-SUMMARY.md (§5.B and §5.C).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.