amirbena / amirbena/code-review-skill
Benchmark-backed behavioral proof for specialist-depth progressive loading
- Dominant language
- Python
- Stars
- 7
- Forks
- 2
- Avg merge
- 27m
- Merged PRs (30d)
- 188
Description
## Problem
Nothing yet proves specialist-depth's conditional loading (#410) is safe
and behaviorally stable. A refactor that quietly changes findings on real
inputs is worse than not refactoring, and the roadmap explicitly requires
proof before any further capability is touched.
## Goal
Benchmark evidence exists, using the repository's existing benchmark
harness (no parallel evaluator), proving: ordinary reviews where
specialist-depth is unnecessary see reduced instruction/resource surface
and unchanged findings; reviews where it must activate see it activate
correctly; an ambiguous/risky activation case behaves conservatively
(loads); and behavior otherwise matches the #408 baseline.
## Scope
- Add or extend a benchmark/reference corpus with at least four cases:
(1) ordinary review, specialist-depth not needed; (2) review where it
must activate; (3) ambiguous/risky activation case; (4) a direct
before/after surface-reduction comparison against #408's baseline.
- Reuse the existing specialist-depth corpora and routing-decision corpus
from #409 rather than duplicating them.
- Record whether findings/verdicts changed versus the #408 baseline.
## Non-Goals
- Rewriting any existing pinned benchmark expectation to make the
refactor pass. Any intentional behavioral contract change found here
must be raised and justified as its own issue, not folded in silently.
- Proving anything about capabilities other than specialist-depth.
## Acceptance Criteria
- [ ] All four required benchmark cases exist and pass
- [ ] Findings/verdicts on the "unnecessary" and "must activate" cases match the #408 baseline
- [ ] The ambiguous case is shown to load (fail-closed), not skip
- [ ] Instruction/resource surface measurably decreases on the "unnecessary" case relative to #408's baseline
- [ ] No existing benchmark expectation was changed to force a pass
## Dependencies
Depends on: #403 (parent), #410, #408
Blocks: none
Parent: #403
## Validation
- Full benchmark run over the new + existing specialist-depth corpora
- Diff of loaded-capability/instruction-surface metrics against #408's baseline
Release category: none - benchmark/test evidence only.
Contributor guide
Assessment
This issue has not been assessed yet.