amirbena / amirbena/code-review-skill

Build shared corpus-hygiene test infrastructure for benchmark and review fixture corpora

Open
#398 0 comments 0 reactions 1 assignee Claimed by @amirbena View on GitHub
area:packaging-portability contributor-owned help wanted priority:P2 type:refactor
Dominant language
Python
Stars
7
Forks
2
Avg merge
27m
Merged PRs (30d)
188

Description

## Type

Refactor

## Area

Packaging / Portability

## Priority

P2 — Medium

Contributor-owned: pure test-infrastructure work; no fixture content, reference model, or corpus schema changes.

## Problem

20 files — 15 under `tests/unit/benchmark/*_corpus.py` and 5 under `tests/unit/review/{root_cause,repository_intelligence,specialist_depth}/*_corpus.py` — total 491 tests, each repeating the same generic ~15–20-test "corpus shape" check against a different `CORPUS_DIR` constant: directory exists with a README, sub-corpus stays small, every fixture file parses and validates, filename stem matches case id, case ids are unique, every case records a rationale and tags, every case pins an explicit consistent decision, patch-case anchors occur in the diff/base, no finding carries a severity outside P0/P1/P2, and README links every fixture / states the selection principle / uses the single reference validator. Only each corpus's own semantic assertions (which cases exist, what they specifically assert) are genuinely per-corpus.

## Goal

One shared, parametrizable corpus-hygiene test base (mixin or equivalent) that every corpus file's generic-shape checks delegate to, while every corpus keeps its own domain-specific semantic assertions in its owning file.

## Scope

- Design one shared corpus-hygiene test base/mixin covering the ~15–20 generic checks, parametrized by `CORPUS_DIR` (and any other per-corpus config already present, e.g. required tags or size bounds).
- Migrate all 20 files to use it for their generic-shape assertions only.
- Leave every corpus-specific semantic assertion (e.g. domain-specific case-count expectations, "all flagged cases are distinct defect kinds") in its owning file, unparametrized.

## Non-Goals

- Does not redesign the `benchmark-case/v1` fixture schema, the benchmark matcher, the finding model, or any `tests/reference/**` reference model — infrastructure only.
- Does not change which fixture files exist or their content.
- Does not merge the `tests/unit/benchmark/` and `tests/unit/review/` corpus trees into one directory — each stays under its current package.

## Acceptance Criteria

- [ ] Baseline test count for the 20 scoped files recorded (491).
- [ ] After-change test count recorded and reported.
- [ ] Every generic-shape check still runs, per corpus; no semantic assertion silently dropped.
- [ ] Corpus-specific content stays locally owned and unparametrized.
- [ ] A failing case still identifies which corpus/file broke.
- [ ] Scoped tests and the full suite (`python3 -m unittest discover -s tests -t .`) pass.

## Dependencies

Depends on: none
Blocks: none
Parent: #395

Independent of the other four children (disjoint files, no shared-helper collision) — can be implemented in parallel.

## Validation

- Before/after: `grep -rc "def test_" tests/unit/benchmark/*_corpus.py tests/unit/review/{root_cause,repository_intelligence,specialist_depth}/*_corpus.py`.
- `python3 -m unittest discover -s tests -t .`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.