amirbena / amirbena/code-review-skill

Consolidate tests/policy/benchmark/ doc-pinning skeletons

Open
#397 0 comments 0 reactions 1 assignee Claimed by @amirbena View on GitHub
area:packaging-portability contributor-owned help wanted priority:P2 type:refactor
Dominant language
Python
Stars
7
Forks
2
Avg merge
27m
Merged PRs (30d)
188

Description

## Type

Refactor

## Area

Packaging / Portability

## Priority

P2 — Medium

Contributor-owned: bounded to `tests/policy/benchmark/`, no benchmark design-doc content or reference-model semantics touched.

## Problem

`tests/policy/benchmark/`'s 8 `*_docs.py` files (145 tests) each pin a different `docs/benchmark/*.md` design record, but ~9–11 of each file's tests are the identical structural skeleton (not packaged/repository-development-only, names its issue and neighbours, canonical invariant stated verbatim, determinism rules explicit, worked examples are the conformance bar, scope boundaries defer neighbours, status defers to an eventual canonical home, README/ARCHITECTURE wiring, reference module declared test-only, unit test consumes the single reference module) applied to a different doc path and literal phrase text per file.

## Goal

Replace the repeated structural skeleton with one parametrized/table-driven check over `(doc_path, canonical_invariant_text, section_headings)`, while each doc's actual canonical-invariant text and doc-specific structural content remains individually verified and independently attributable on failure.

## Scope

- Extract the shared ~9–11-test structural skeleton into parametrized infrastructure over the 8 benchmark design docs.
- Keep each doc's unique canonical-invariant text and any doc-specific structural sections as distinct, locally-owned assertions.
- Preserve doc-specific failure attribution.

## Non-Goals

- Does not touch `test_benchmark_ci_classifier_boundary.py` (same directory, a different pattern — an AST-based import-boundary guard, not doc-pinning).
- Does not change any `docs/benchmark/*.md` content, the benchmark reference models, or the documented benchmark/reference-model independence.

## Acceptance Criteria

- [ ] Baseline test count for the 8 scoped files recorded (145).
- [ ] After-change test count recorded and reported.
- [ ] Every invariant currently asserted remains checked somewhere; no semantic assertion silently dropped.
- [ ] Doc-specific content stays locally owned, not folded into the generic skeleton.
- [ ] A failing parametrized case still identifies which doc broke.
- [ ] Scoped tests and the full suite (`python3 -m unittest discover -s tests -t .`) pass.

## Dependencies

Depends on: none
Blocks: none
Parent: #395

## Validation

- Before/after: `grep -rc "def test_" tests/policy/benchmark/test_*_docs.py`.
- `python3 -m unittest discover -s tests -t .`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.