amirbena / amirbena/code-review-skill
Consolidate tests/policy/benchmark/ doc-pinning skeletons
- Dominant language
- Python
- Stars
- 7
- Forks
- 2
- Avg merge
- 27m
- Merged PRs (30d)
- 188
Description
## Type
Refactor
## Area
Packaging / Portability
## Priority
P2 — Medium
Contributor-owned: bounded to `tests/policy/benchmark/`, no benchmark design-doc content or reference-model semantics touched.
## Problem
`tests/policy/benchmark/`'s 8 `*_docs.py` files (145 tests) each pin a different `docs/benchmark/*.md` design record, but ~9–11 of each file's tests are the identical structural skeleton (not packaged/repository-development-only, names its issue and neighbours, canonical invariant stated verbatim, determinism rules explicit, worked examples are the conformance bar, scope boundaries defer neighbours, status defers to an eventual canonical home, README/ARCHITECTURE wiring, reference module declared test-only, unit test consumes the single reference module) applied to a different doc path and literal phrase text per file.
## Goal
Replace the repeated structural skeleton with one parametrized/table-driven check over `(doc_path, canonical_invariant_text, section_headings)`, while each doc's actual canonical-invariant text and doc-specific structural content remains individually verified and independently attributable on failure.
## Scope
- Extract the shared ~9–11-test structural skeleton into parametrized infrastructure over the 8 benchmark design docs.
- Keep each doc's unique canonical-invariant text and any doc-specific structural sections as distinct, locally-owned assertions.
- Preserve doc-specific failure attribution.
## Non-Goals
- Does not touch `test_benchmark_ci_classifier_boundary.py` (same directory, a different pattern — an AST-based import-boundary guard, not doc-pinning).
- Does not change any `docs/benchmark/*.md` content, the benchmark reference models, or the documented benchmark/reference-model independence.
## Acceptance Criteria
- [ ] Baseline test count for the 8 scoped files recorded (145).
- [ ] After-change test count recorded and reported.
- [ ] Every invariant currently asserted remains checked somewhere; no semantic assertion silently dropped.
- [ ] Doc-specific content stays locally owned, not folded into the generic skeleton.
- [ ] A failing parametrized case still identifies which doc broke.
- [ ] Scoped tests and the full suite (`python3 -m unittest discover -s tests -t .`) pass.
## Dependencies
Depends on: none
Blocks: none
Parent: #395
## Validation
- Before/after: `grep -rc "def test_" tests/policy/benchmark/test_*_docs.py`.
- `python3 -m unittest discover -s tests -t .`
Contributor guide
Assessment
This issue has not been assessed yet.