amirbena / amirbena/code-review-skill

Add a mechanical citation-existence fidelity check to the benchmark harness

Open
#349 0 comments 0 reactions 1 assignee Claimed by @amirbena View on GitHub
area:review-quality maintainer-led priority:P2 type:quality
Dominant language
Python
Stars
7
Forks
2
Avg merge
27m
Merged PRs (30d)
188

Description

## Type

Quality

## Area

Review Quality

## Priority

P2 — Medium

## Problem

A finding's cited file/line/snippet being accurate is a cheaper, different claim than being "inspected" — and nothing today checks even that cheaper claim mechanically. The benchmark harness could catch a fabricated-but-plausible citation (wrong file, wrong line, a snippet never actually present at the reviewed SHA) automatically, without needing `#182`'s telemetry at all, but no such check exists. This only becomes meaningful once `#342` lands, since the harness does not yet carry a produced finding's real location/symbol data through to any consumer.

## Goal

Add a benchmark-harness check that flags a produced finding whose cited location does not exist, or whose quoted evidence snippet is not actually present near that location, at the reviewed SHA.

## Scope

- A new metric/check in the benchmark pipeline that, for each produced finding, verifies the cited file exists at the reviewed SHA and the line/symbol is in range.
- Best-effort verification that quoted evidence text is actually present near the cited location (bounded substring/fuzzy match).
- Report fabricated-citation findings as a distinct metric category, separate from match/near-miss/no-match against fixtures.
- Wire the check into the existing benchmark run/report output alongside the existing quality metrics (#55/#56/#57).

## Non-Goals

- Does not prove the citation was actually inspected — existence only (see the deferred citation-grounding cross-check).
- Not a runtime gate on live reviews — benchmark/CI signal only.
- Does not change severity, confidence, identity, or the decision derivation for any finding.
- Does not touch the matcher's fixture-correspondence logic (`match-criteria.md`) — this is a separate, additional check.

## Acceptance Criteria

- [ ] Running the benchmark corpus flags a synthetic fixture with a deliberately fabricated location/snippet.
- [ ] Running the benchmark corpus does not flag any existing known-good produced finding once `#342` has landed.
- [ ] The new metric is reported separately from match/near-miss/no-match outcomes.
- [ ] A regression test covers both a genuine and a fabricated citation.

## Dependencies

Depends on: #342 (needs real location/symbol/claim data threaded through the harness first)
Relates: #54, #55, #56, #57

## Validation

- New unit/regression tests pass.
- A scripted run against at least one deliberately-fabricated fixture demonstrates detection.

## Canonical design

Part of the architecture defined in
[`docs/benchmark-measurement-architecture/benchmark-measurement-architecture-model.md`](docs/benchmark-measurement-architecture/benchmark-measurement-architecture-model.md),
§12.3 ("Benchmark citation-fidelity signal"). Read the canonical design
before implementation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.