New skill: /review-audit — compare deep-review output against PR comments to find blind spots
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 1
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 7
Description
Summary
Create a /review-audit skill that automates the comparison of /deep-review output
against actual PR review comments, producing a gap analysis, root cause classification,
and CLAUDE.md improvement recommendations.
Motivation
Two manual review-audit sessions (PR 4366, PR 4364) demonstrated high value but required
~15 tool calls and significant synthesis each time. The workflow is:
- Find the PR for the current branch (ADO or GitHub)
- Pull all review threads and comments
- Parse human findings into a structured list
- Map each against the deep-review output (overlap, missed, unique)
- Classify root causes for misses
- Propose CLAUDE.md additions to prevent recurrence
Both sessions produced actionable CLAUDE.md improvements and identified systematic skill
blind spots. This should be a one-command skill.
Proposed interface
/review-audit [PR-number-or-URL]
If no PR is specified, auto-detect from the current branch's open PR.
Expected output
Comparison table
- Both caught: findings surfaced by both the automated review and human reviewer
- PR only: findings the human reviewer caught but
/deep-reviewmissed - Deep-review only: findings
/deep-reviewcaught but the human reviewer missed
Root cause classification (for PR-only findings)
For each finding the automated review missed, classify why:
- Missing domain knowledge (needs CLAUDE.md addition)
- Agent prompt blind spot (needs deep-review prompt update)
- Severity miscalibration (caught but undergraded)
- Out of scope (pre-existing issue, not introduced by the PR)
Scorecard
- Hit rate (% of human findings also caught by automated review)
- Miss count by severity (critical/warning/suggestion)
- Unique finds per side
Recommendations
- Specific CLAUDE.md additions to prevent recurrence (ask user before applying)
- Specific deep-review prompt improvements (reference skill issue if exists)
Matching logic
The skill must handle:
- Paraphrased findings: human says "race on feature entry", automated says "render gate
shows calculator during error state" — same root cause, different framing - Different file:line references: same issue pointed at different lines
- Severity disagreements: human says blocking, automated says suggestion — flag as
severity miscalibration - Partial overlap: human finding covers A+B, automated finding covers only A — flag B
as a gap
Acceptance criteria
- Skill fetches PR review threads from ADO (or GitHub, based on repo remote)
- Skill reads the most recent
/deep-reviewoutput from the conversation context - Produces structured comparison: Both caught, PR only, Deep-review only
- For each PR-only miss, classifies root cause
- Outputs scorecard with hit rate and miss counts
- Proposes CLAUDE.md additions for knowledge gaps
- Asks user before writing any changes
- Works for PRs targeting any branch (not just main)
Why a skill, not documentation?
The value is in the automated comparison and structured output. The matching logic needs to
handle paraphrased findings, different file:line references, and severity disagreements
across two unstructured formats. This is inherently procedural and benefits from tool
orchestration that cannot be captured in a documentation rule.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the proposed /review-audit interface and the existing /deep-review conversation output, then trace how PR review threads are fetched from ADO or GitHub. Implement the comparison, root-cause classification, scorecard, recommendations, and confirmation before changes described in the acceptance criteria. Done means it works for PRs targeting any branch and produces the three requested finding groups.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, python
- Domain
- developer-experience, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100