TimZander / TimZander/claude

New skill: /review-audit — compare deep-review output against PR comments to find blind spots

Open
#91 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
1
Avg merge
1d 3h
Merged PRs (30d)
7

Description

Summary

Create a /review-audit skill that automates the comparison of /deep-review output
against actual PR review comments, producing a gap analysis, root cause classification,
and CLAUDE.md improvement recommendations.

Motivation

Two manual review-audit sessions (PR 4366, PR 4364) demonstrated high value but required
~15 tool calls and significant synthesis each time. The workflow is:

  1. Find the PR for the current branch (ADO or GitHub)
  2. Pull all review threads and comments
  3. Parse human findings into a structured list
  4. Map each against the deep-review output (overlap, missed, unique)
  5. Classify root causes for misses
  6. Propose CLAUDE.md additions to prevent recurrence

Both sessions produced actionable CLAUDE.md improvements and identified systematic skill
blind spots. This should be a one-command skill.

Proposed interface

/review-audit [PR-number-or-URL]

If no PR is specified, auto-detect from the current branch's open PR.

Expected output

Comparison table
  • Both caught: findings surfaced by both the automated review and human reviewer
  • PR only: findings the human reviewer caught but /deep-review missed
  • Deep-review only: findings /deep-review caught but the human reviewer missed
Root cause classification (for PR-only findings)

For each finding the automated review missed, classify why:

  • Missing domain knowledge (needs CLAUDE.md addition)
  • Agent prompt blind spot (needs deep-review prompt update)
  • Severity miscalibration (caught but undergraded)
  • Out of scope (pre-existing issue, not introduced by the PR)
Scorecard
  • Hit rate (% of human findings also caught by automated review)
  • Miss count by severity (critical/warning/suggestion)
  • Unique finds per side
Recommendations
  • Specific CLAUDE.md additions to prevent recurrence (ask user before applying)
  • Specific deep-review prompt improvements (reference skill issue if exists)

Matching logic

The skill must handle:

  • Paraphrased findings: human says "race on feature entry", automated says "render gate
    shows calculator during error state" — same root cause, different framing
  • Different file:line references: same issue pointed at different lines
  • Severity disagreements: human says blocking, automated says suggestion — flag as
    severity miscalibration
  • Partial overlap: human finding covers A+B, automated finding covers only A — flag B
    as a gap

Acceptance criteria

  • Skill fetches PR review threads from ADO (or GitHub, based on repo remote)
  • Skill reads the most recent /deep-review output from the conversation context
  • Produces structured comparison: Both caught, PR only, Deep-review only
  • For each PR-only miss, classifies root cause
  • Outputs scorecard with hit rate and miss counts
  • Proposes CLAUDE.md additions for knowledge gaps
  • Asks user before writing any changes
  • Works for PRs targeting any branch (not just main)

Why a skill, not documentation?

The value is in the automated comparison and structured output. The matching logic needs to
handle paraphrased findings, different file:line references, and severity disagreements
across two unstructured formats. This is inherently procedural and benefits from tool
orchestration that cannot be captured in a documentation rule.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the proposed /review-audit interface and the existing /deep-review conversation output, then trace how PR review threads are fetched from ADO or GitHub. Implement the comparison, root-cause classification, scorecard, recommendations, and confirmation before changes described in the acceptance criteria. Done means it works for PRs targeting any branch and produces the three requested finding groups.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, python
Domain
developer-experience, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.