anthropics / anthropics/claude-code-action

Workflow-validation skip reports conclusion: success — indistinguishable from a real pass (follow-up to #1343)

Offen
#1,632 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug dev-experience p2
Vorherrschende Sprache
TypeScript
Sterne
8.9k
Forks
2.1k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

Follow-up to #1343, which unified the two validation-skip code paths on `success`. That fixed the inconsistency, but the chosen value has a security-relevant side effect: **a skipped review is now indistinguishable from a passing review** — including to branch protection.

#1343 itself proposed the safer alternative in its final paragraph ("both paths could surface as `skipped` (or `neutral`)"). This issue argues that alternative should be preferred.

## Observed

Confirmed on `claude-code-action@v1`, private repo, August 2026 — both the first-add and modification paths now return `success`:

| Run | PR changed | Duration | Conclusion | Log contains |
|---|---|---|---|---|
| `31280931811` | adds a workflow | 14s | `success` | `Exiting due to workflow validation skip` |
| `31448510042` | **modifies** `claude-review.yml` | 16s | `success` | `Exiting due to workflow validation skip` |

For contrast, a genuine review run on the same repo takes ~2m39s–5m39s and reports real `num_turns` / `total_cost_usd`.

## Why `success` is the wrong value here

The anti-tamper check itself is sound — a PR should not be able to rewrite the reviewer that gates it. The problem is only the conclusion it reports.

**Any PR that edits the invoking workflow gets a green `claude-review` check with nothing reviewed — including a PR that weakens or disables the reviewer.** The check is structurally incapable of failing on exactly the PRs where review matters most.

Because the conclusion is `success`, this is invisible to:
- the checks UI (a green ✓ identical to a real pass)
- **branch protection** — `claude-review` as a required status check is satisfied by a run that never executed
- any automation gating on check conclusions (e.g. Dependabot auto-merge)

The only way to distinguish the two today is to open the run log and grep for `Exiting due to workflow validation skip`, or notice the ~10x duration difference. We were caught by this in practice: a workflow-fix PR showed green, and it took reading the log to establish that the check had done nothing.

## Suggested fix

Report the skip as **`neutral`** (GitHub's conclusion for "ran but intentionally reached no verdict") rather than `success`.

This preserves everything #1343 asked for — no spurious red ✗, both paths consistent, no "something is broken" impression — while keeping the outcome distinguishable from a real pass. Concretely, `neutral`:

- does **not** satisfy a required status check, so a workflow-editing PR can no longer merge on the strength of a review that never ran
- renders distinctly in the checks UI
- is already the conventional signal for "skipped by design"

If `neutral` is considered too disruptive for existing users relying on the green check, a lower-impact alternative would be to keep `success` but set a distinguishable check **output/summary** (e.g. title "Skipped — workflow modified") so tooling has something to key on without reading raw logs.

## Reproduction

1. Add `claude-code-action@v1` to a repo and merge it.
2. Open a PR that modifies that workflow file (change `--max-turns`, say).
3. Observe: check reports `success` in seconds; the log contains `Exiting due to workflow validation skip`; no review was performed.
4. Make `claude-review` a required status check and observe that the PR is mergeable on the strength of that skip.

## Workaround for others hitting this

Until upstream changes, a separate workflow filtered on the invoking workflow's path can post a notice on affected PRs. Worth noting the constraint that makes this awkward: the action cannot be fixed or tested from a PR that edits it, because that PR self-skips — the change has to land on the default branch first and be exercised on an unrelated PR.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.