amirbena / amirbena/code-review-skill
Add a mechanical citation-existence fidelity check to the benchmark harness
- Dominant language
- Python
- Stars
- 7
- Forks
- 2
- Avg merge
- 27m
- Merged PRs (30d)
- 188
Description
## Type
Quality
## Area
Review Quality
## Priority
P2 — Medium
## Problem
A finding's cited file/line/snippet being accurate is a cheaper, different claim than being "inspected" — and nothing today checks even that cheaper claim mechanically. The benchmark harness could catch a fabricated-but-plausible citation (wrong file, wrong line, a snippet never actually present at the reviewed SHA) automatically, without needing `#182`'s telemetry at all, but no such check exists. This only becomes meaningful once `#342` lands, since the harness does not yet carry a produced finding's real location/symbol data through to any consumer.
## Goal
Add a benchmark-harness check that flags a produced finding whose cited location does not exist, or whose quoted evidence snippet is not actually present near that location, at the reviewed SHA.
## Scope
- A new metric/check in the benchmark pipeline that, for each produced finding, verifies the cited file exists at the reviewed SHA and the line/symbol is in range.
- Best-effort verification that quoted evidence text is actually present near the cited location (bounded substring/fuzzy match).
- Report fabricated-citation findings as a distinct metric category, separate from match/near-miss/no-match against fixtures.
- Wire the check into the existing benchmark run/report output alongside the existing quality metrics (#55/#56/#57).
## Non-Goals
- Does not prove the citation was actually inspected — existence only (see the deferred citation-grounding cross-check).
- Not a runtime gate on live reviews — benchmark/CI signal only.
- Does not change severity, confidence, identity, or the decision derivation for any finding.
- Does not touch the matcher's fixture-correspondence logic (`match-criteria.md`) — this is a separate, additional check.
## Acceptance Criteria
- [ ] Running the benchmark corpus flags a synthetic fixture with a deliberately fabricated location/snippet.
- [ ] Running the benchmark corpus does not flag any existing known-good produced finding once `#342` has landed.
- [ ] The new metric is reported separately from match/near-miss/no-match outcomes.
- [ ] A regression test covers both a genuine and a fabricated citation.
## Dependencies
Depends on: #342 (needs real location/symbol/claim data threaded through the harness first)
Relates: #54, #55, #56, #57
## Validation
- New unit/regression tests pass.
- A scripted run against at least one deliberately-fabricated fixture demonstrates detection.
## Canonical design
Part of the architecture defined in
[`docs/benchmark-measurement-architecture/benchmark-measurement-architecture-model.md`](docs/benchmark-measurement-architecture/benchmark-measurement-architecture-model.md),
§12.3 ("Benchmark citation-fidelity signal"). Read the canonical design
before implementation.
Contributor guide
Assessment
This issue has not been assessed yet.