devondragon / devondragon/ccmagic
review: QUICK path returns PASS WITH WARNINGS when only INFO findings exist
Nobody has claimed this yet.
- Dominant language
- Shell
- Stars
- 6
- Forks
- 0
- Avg merge
- 28m
- Merged PRs (30d)
- 5
Description
Observed
In the evals/ review suite pilot (2026-09-18), case 01-quick-clean-log-fix (a one-file diff that rewords a comment and a debug log line) routed to QUICK, raised a single INFO finding, and closed with:
PASS WITH WARNINGS — no critical or correctness-breaking issues; one INFO-level nit on log/output consistency.
A second pilot run on the same prompt returned PASS, so the behavior is intermittent.
Expected
skills/review/SKILL.md defines the QUICK verdicts as:
- PASS: no critical or warning findings
- PASS WITH WARNINGS: warnings only
- FAIL: one or more critical findings
INFO-only output should be PASS.
Suggested fix
Make the verdict rule explicit right above the verdict line in the QUICK output section, for example: "Count only CRITICAL and WARNING findings when choosing the verdict. INFO findings never change it."
How to verify
Run the suite; grader no-false-findings on case 01 fails when this happens.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with skills/review/SKILL.md and inspect the QUICK verdict rules around the output section. Run the review suite, especially grader no-false-findings on case 01, and confirm that INFO-only findings produce PASS while warning findings still produce PASS WITH WARNINGS.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- shell
- Domain
- testing-qa, tooling
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100