Codewhale review gate fails closed on large diffs: reasoning consumes the whole output budget, review never posts
- Dominant language
- Rust
- Stars
- 41k
- Forks
- 3.6k
- Avg merge
- 13h 59m
- Merged PRs (30d)
- 299
Description
## What happens
The `Codewhale review` required check fails on PRs with large diffs, and it fails
*without producing a review* — so a required gate blocks the merge while telling
the author nothing about their code.
Observed on PR #6281 and PR #6284 (both currently blocked by this, both with
otherwise-green required checks):
```
Review key: BYOK provider secret (provider: deepseek)
Review limits: 200000 characters per pass, at most 1 passes
Output budget: CLI automatic cap (no override set)
error: Review pass 1/1 incomplete: provider stop reason `length`;
the partial review was not accepted or posted.;
publication: not_attempted; completed review passes: 0/1;
accumulated usage: {"input_tokens":35839,"output_tokens":65536,
"prompt_cache_hit_tokens":640,"prompt_cache_miss_tokens":35199,
"reasoning_tokens":65536}
```
Run: https://github.com/Hmbown/Codewhale/actions/runs/35069645353
## The actual defect
`output_tokens: 65536` and `reasoning_tokens: 65536` are the same number. The
model spent its **entire** output allowance on reasoning and emitted zero
review text, then hit `length` and the action discarded the (empty) partial.
So the budget is being accounted in the wrong unit for a reasoning model: the
cap has to leave room for the answer *after* reasoning, or reasoning has to be
bounded separately. This is the same shape as the sub-agent hand-back reserve
problem in #6194 — a reserve sized as if only the visible output costs
anything.
Two independent things are wrong:
1. **It fails closed with no signal.** A gate that blocks a merge must say why
the *code* is wrong. "The reviewer ran out of budget" is the gate's problem,
not the author's, and it should not read as a failed review. Either surface
it as a distinct neutral/skipped outcome, or retry with a reduced scope.
2. **One pass, no degradation path.** `at most 1 passes` with a 200k-character
input and no chunking means the gate gets *less* reliable exactly as a PR
gets bigger — the case where review matters most.
## Acceptance criteria
- [ ] A reasoning-model review reserves output capacity for the review text
rather than letting reasoning consume the whole cap; the reserve is sized
off the model's reasoning behaviour, not a flat constant
- [ ] A budget exhaustion is reported as an infrastructure outcome distinct
from "this PR failed review", and does not silently block a merge with no
findings
- [ ] A too-large diff degrades (chunked passes, or reviews the highest-risk
files and says what it skipped) instead of returning nothing
- [ ] No silent caps: whatever the gate did not read, it names
## Notes
Copilot and Codex reviewers on the same PRs are also unavailable (quota), so
this gate is currently the only automated reviewer, which raises the cost of it
failing closed.
Contributor guide
Research direction
Start with the Codewhale review gate and the failing Actions run 35069645353, comparing the reported input, output, and reasoning token usage. Trace how the gate handles budget exhaustion, partial reviews, publication, and oversized diffs. Done means reasoning reviews retain answer capacity, exhaustion is distinct from review failure, large diffs degrade visibly, and skipped work is named.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ai, ci-cd, devtools
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100