devondragon / devondragon/ccmagic

review: QUICK path returns PASS WITH WARNINGS when only INFO findings exist

Open Beginner friendly
#36 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Shell
Stars
6
Forks
0
Avg merge
28m
Merged PRs (30d)
5

Description

Observed

In the evals/ review suite pilot (2026-09-18), case 01-quick-clean-log-fix (a one-file diff that rewords a comment and a debug log line) routed to QUICK, raised a single INFO finding, and closed with:

PASS WITH WARNINGS — no critical or correctness-breaking issues; one INFO-level nit on log/output consistency.

A second pilot run on the same prompt returned PASS, so the behavior is intermittent.

Expected

skills/review/SKILL.md defines the QUICK verdicts as:

  • PASS: no critical or warning findings
  • PASS WITH WARNINGS: warnings only
  • FAIL: one or more critical findings

INFO-only output should be PASS.

Suggested fix

Make the verdict rule explicit right above the verdict line in the QUICK output section, for example: "Count only CRITICAL and WARNING findings when choosing the verdict. INFO findings never change it."

How to verify

Run the suite; grader no-false-findings on case 01 fails when this happens.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with skills/review/SKILL.md and inspect the QUICK verdict rules around the output section. Run the review suite, especially grader no-false-findings on case 01, and confirm that INFO-only findings produce PASS while warning findings still produce PASS WITH WARNINGS.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
testing-qa, tooling
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.