anthropics / anthropics/claude-code-action

Action posts repo's CLAUDE.md verbatim as comment body instead of structured review (workflow exits success)

Open
#1,302 0 comments 0 reactions 0 assignees View on GitHub
bug experimental-review p2 provider:1p
Dominant language
TypeScript
Stars
8.9k
Forks
2.1k
Avg merge
3d 9h
Merged PRs (30d)
10

Description

## Summary

`anthropics/claude-code-action@v1` **intermittently posts the repository's `CLAUDE.md` file verbatim** as its PR comment, instead of producing the structured review the workflow's `prompt:` requests. The action exits `conclusion: success`, so the workflow check is green, but no real review happened.

## Environment

- Action: `anthropics/claude-code-action@v1` (floating tag — version drift may be relevant)
- Trigger: `pull_request` (synchronize event on a human-pushed commit)
- Auth: `claude_code_oauth_token`
- Model: action-selected default

## Repro

Same workflow ran on two consecutive commits of the same PR. First run = normal structured review. Second run = CLAUDE.md echo.

| Run ID | Conclusion | Comment shape |
|---|---|---|
| `25720844923` (initial) | `success` | 676-char structured security review, ends with the configured verdict-tag HTML comment |
| `25722050413` (1-line YAML follow-up) | `success` | **19,407-char raw CLAUDE.md** verbatim, no verdict tag |

First 120 chars of the misfire comment body:

\`\`\`
# CLAUDE.md

Guidance for Claude Code (claude.ai/code) working in this repository.

> **User context** lives in `USER.md` (working
\`\`\`

This is the exact opening of our repo's `CLAUDE.md` file (top-level guidance file — see https://docs.claude.com/en/docs/claude-code/memory).

## Workflow config (security-review.yml)

\`\`\`yaml
- uses: anthropics/claude-code-action@v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
github_token: ${{ secrets.GITHUB_TOKEN }}
show_full_output: true
prompt: |
Perform a security review of this pull request. ...
(template requests structured output with severity classification + verdict tag)
claude_args: '--allowed-tools "Bash(gh pr view:*),Bash(gh pr diff:*),Bash(gh pr comment:*)"'
\`\`\`

Note: `claude_args` only allows `gh pr view/diff/comment`. The model **cannot directly `cat CLAUDE.md`** under these tool restrictions — so the file's content must be reaching the model via the action's own internal context-loading step, not via a model-issued tool call.

## Hypothesis

`CLAUDE.md` is auto-loaded into the model's context (per `--memory` semantics of Claude Code, which the action wraps). Intermittently, the model conflates **content loaded as guidance** with **content to post**, and pipes CLAUDE.md verbatim into `gh pr comment`. The structured review template is silently skipped.

The 1-line YAML-comment-only diff between the two commits suggests this is not content-triggered by the PR diff itself — it's a flake in the action's prompt-wrapper layer.

## Expected

Either:
1. The action produces the structured review the `prompt:` template requested, OR
2. The action fails the run with `conclusion: failure` if the model output is non-conforming.

It should not exit success with an unrelated dump.

## Impact

- **Self-detecting setups** (workflows with verdict-tag fail-safes that parse the final line for a binding token) correctly skip auto-approve — no bad approvals ship.
- **Naive setups** (workflows that treat green check = review happened) silently accept a non-review.
- Wastes workflow minutes + OAuth quota when retried.

## Suggested fixes

1. The action could detect "model output is suspiciously similar to a memory file it loaded" (≥10kb body whose head matches a loaded guidance file) and retry-or-fail rather than post.
2. The action could separate the model's "guidance context" (read-only memory) from "documents available for tool calls" (Bash/Read targets), so guidance is never re-emittable as content.
3. Strengthen the prompt-template scaffolding around the review-output schema (e.g. require a tagged opening line) so non-conforming output is rejected at the action layer, not by the consumer's parser.

## Workaround

Re-trigger via `workflow_dispatch` (or via a `@claude review please` comment on workflows that use the separate `claude.yml` pattern). Both yielded the correct structured output on retry — supports the "flake, not deterministic" interpretation.

Happy to provide additional run logs or correlate further with action-side telemetry if useful.

Contributor guide

Open the contributing guide

Research direction

Start with the security-review.yml workflow and compare runs 25720844923 and 25722050413, including their logs and posted comment bodies. Trace the action's prompt-wrapper and internal context-loading behavior, then verify that non-conforming output such as a raw CLAUDE.md dump is rejected or causes a failed run instead of being posted as a successful review.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
ci-cd, devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.