ColoredCow / ColoredCow/engineering-recipes
Code review comment posting is nondeterministic — recipe prompt should explicitly require gh pr comment
- Dominant language
- Shell
- Stars
- 1
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
The shared `claude-code-review.yml` reusable workflow **posts the review comment only intermittently**. The review almost always runs to completion, but whether the result actually lands as a PR comment is **nondeterministic** — it depends on whether Claude *chooses* to call `gh pr comment` itself, which is driven by an implicit instruction in the recipe's default prompt.
We hit this in `DostEducation/dost-ai-voicebot`: the workflow ran green, but no review comment appeared on the PR. Meanwhile other repos on the same recipe (`DostEducation/admindashboard`, etc.) do get the comment — which initially looked like a per-repo misconfiguration but is actually the same latent flakiness.
## Root cause
The recipe runs `anthropics/claude-code-action@v1.0.8` in **`agent` mode** (auto-detected for `pull_request` events) with **`track_progress: false`** (it's commented out in the recipe because it's incompatible with the `labeled` trigger consumers use).
In `agent` mode with `track_progress` off, **the action does not auto-post Claude's output as a PR comment.** Posting becomes Claude's own responsibility — it must run `gh pr comment` (allowlisted via the default `Bash(gh pr comment:*)` tool).
The default prompt only says:
> **Review Output Format - IMPORTANT**
> Create ONE single comment with your complete review using this structure: ...
> Do NOT create multiple inline comments. Put everything in ONE comment.
This describes the *format* of the comment but never explicitly tells Claude to **run `gh pr comment` to post it**. So Claude sometimes interprets "create one comment" as "actually post via the tool" and sometimes as "produce one block of text" and stops. The result is flaky.
## Evidence
**Working run — `DostEducation/admindashboard` PR #1935** (run `28353530110`, `agent` mode, default recipe prompt):
- Log contains: `"text": "Now let me post my review as a single comment:"` followed by a `Bash` tool call.
- PR #1935 has `claude`-authored `## Code Review Summary` comments. ✅ Posted.
**Failing run — `DostEducation/dost-ai-voicebot` PR #22** (`agent` mode, identical recipe prompt):
- Claude emitted the full review as its **final assistant text** and **never called `gh pr comment`**.
- No review comment on the PR (only the unrelated pytest-coverage comment). ❌ Not posted.
- Likely aggravated by an unusually large/truncated diff that nudged Claude to "wrap up" by printing rather than posting — but the underlying cause is the implicit prompt either way.
Same recipe (`@main`), same `claude-code-action@v1.0.8`, same `agent` mode, same model (`claude-sonnet-4-5`), byte-identical consumer workflow files. The only difference is whether Claude decided to post.
## Affected repos
All consumers of `ColoredCow/engineering-recipes/.github/workflows/claude-code-review.yml@main`, e.g.:
- `DostEducation/dost-ai-voicebot`
- `DostEducation/admindashboard`
- `ColoredCow/megafitmeals`
- `ColoredCow/goonj`
The ones that "work today" are getting lucky on the flaky prompt and can silently miss reviews too.
## Proposed fix (minimal, correct)
Make the posting step **explicit and mandatory** in the recipe's default prompt. Append a step like:
```
6. **Post the Review (REQUIRED — do not skip)**
You are running headless: your text output is NOT posted automatically.
You MUST publish the review as a single PR comment yourself:
- Write the full review markdown to /tmp/review.md (Write tool), then run:
gh pr comment --body-file /tmp/review.md
The task is not complete until the comment is posted.
```
And add `Write` to the default `allowed_tools` fallback so `--body-file` works:
```
'Read,Write,Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*)'
```
(Inline `--body` works too and needs no `Write`, but `--body-file` avoids shell-escaping large markdown reviews.)
Because the recipe is pinned `@main`, this fixes every consumer with **zero change to any consumer repo** — and therefore no `claude-code-action` OIDC "Workflow validation failed" friction (that error only triggers when a consumer edits its own `code-review.yml`).
## More robust alternative (larger change)
Set `track_progress: true` so the **action** creates/updates a sticky review comment deterministically instead of relying on Claude. This requires switching consumers off the `labeled` trigger to a supported one (`opened` / `synchronize` / `ready_for_review` / `reopened`), since `track_progress` doesn't support `labeled` — that's why it was disabled in the first place. Worth considering as a follow-up, but the explicit-prompt fix above resolves the immediate flakiness with the least disruption.
## Notes / red herring
While debugging we briefly tried fixing this per-repo by overriding the `prompt` input in the consumer's `code-review.yml`. That works functionally, but editing the consumer workflow trips `claude-code-action`'s OIDC token exchange (`401 Workflow validation failed: the workflow file must … have identical content to the version on the repository's default branch`) on the PR that introduces the edit — expected behavior, self-resolves on merge. Fixing the recipe upstream avoids this entirely since consumer workflows stay unchanged.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with .github/workflows/claude-code-review.yml and inspect the default prompt and allowed_tools fallback used by anthropics/claude-code-action@v1.0.8. Make the posting requirement explicit and ensure the needed tool is allowed, then verify the workflow requires a single PR comment rather than only emitting review text.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, github-actions, shell
- Domain
- ci-cd, devops
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100