amirbena / amirbena/code-review-skill

Benchmark/evaluation subsystem: define ownership boundary and relocate out of generic docs (late Tier 5)

Open
#425 0 comments 0 reactions 1 assignee Claimed by @amirbena View on GitHub
area:instruction-architecture maintainer-led priority:P2 type:refactor
Dominant language
Python
Stars
7
Forks
2
Avg merge
27m
Merged PRs (30d)
188

Description

## Problem

`docs/benchmark/` (corpus, taxonomy/index, selector, matcher/correspondence
logic, execution runner, nightly history/baselines, drift detection,
validation) is classified as generic "documentation" purely because it is
Markdown/YAML under `docs/`. `docs/capability-architecture/capability-architecture-model.md`
§H.1 shows the opposite is true structurally: corpus directory names
already match capabilities, corpus↔policy commit overlap is 0 for 6 of 7
capabilities, and corpus is welded to `tests/` (31/31 corpus commits also
touch `tests/`) — a first-class evaluation subsystem with its own
lifecycle, not incidental prose. §E.2's recommended topology already
reflects this: it moves the harness to `platform/benchmark/` and moves
corpus into `capabilities//corpus/`, not `docs/`.

Left unaddressed, this misclassification risks a second failure mode as
progressive/conditional loading (#403) generalizes: docs-routing becomes
the accidental owner of benchmark semantics simply because both currently
live under `docs/`, even though nothing else about the benchmark
subsystem behaves like documentation.

Separately, the project Wiki already carries benchmark/evaluation
architecture pages written against the current, pre-relocation structure.
Once this issue's relocation happens, those pages become a second,
silent source of the same misclassification: they will keep describing
benchmark/evaluation material through generic-documentation framing,
stale paths, and stale ownership language even after the repository
itself no longer does.

## Goal

Define an explicit ownership boundary for the benchmark/evaluation
subsystem — separate from generic documentation — and relocate it only
where that improves the architecture, once both prerequisite
architectures below have stabilized enough to define that boundary
correctly. The relocation must carry its Wiki documentation forward
consistently, not leave it describing the pre-relocation structure.

## Why this matters for capability routing / progressive loading

Progressive/conditional loading (#403 family) activates capabilities
based on declared ownership boundaries. If benchmark/evaluation material
stays implicitly filed under generic `docs/` routing, a future capability
loader has no principled way to treat "benchmark corpus for capability X"
differently from "human-facing explainer for capability X" — the same
docs-routing rules would apply to both by default. This issue makes that
boundary explicit so the capability manifest work already in flight can
eventually recognize benchmark/evaluation material as its own class,
consistent with model.md §E.2's target topology.

## Scheduling — late Tier 5, non-blocking

This is scheduled **late in Tier 5**, after both:

1. **Tier 4 benchmark architecture stabilizes** (the operational
benchmark-quality loop — selection, nightly execution, drift — closes).
2. **Tier 5 capability/progressive-loading architecture stabilizes**
enough to define the right long-term ownership boundary (the
specialist-depth extraction proves out and the reassessment checkpoint
records whether/how the pattern generalizes).

It must **not** block any current Tier 4 benchmark work, and current Tier
4 work must not wait on it. No implementation happens under this issue
until both prerequisites close.

## Scope (future implementation, once unblocked)

- Inventory every benchmark-owned artifact under `docs/benchmark/`
(corpus, `taxonomy.md`, `corpus-index.json`, selection/matching/
correspondence docs, `runner-contract.md`, `runtime-execution-contract.md`,
`nightly-history-and-baseline.md`, `drift-detection-and-regression-lifecycle.md`)
plus its `scripts/benchmark/*` and `tests/reference/benchmark/*`
counterparts (per model.md §H.3's per-corpus assignment table).
- Define the canonical ownership boundary: which parts are real,
human-facing documentation (stay under `docs/`) versus benchmark/
evaluation infrastructure (relocate).
- Relocate only where it demonstrably improves the architecture — reusing
model.md §E.2's target shape (harness under a platform/benchmark-style
location, corpus alongside its owning capability) rather than inventing
a new topology.
- Update every reference affected by any move: imports, scripts, tests,
the PR-time taxonomy path allowlist (`benchmark_ci_classifier.py` and
its `docs/benchmark/` prefix, flagged in model.md §E.1/§K as needing a
coordinated update per path move), CI workflows, docs navigation, and
capability-manifest routing.
- Update the relevant benchmark/evaluation Wiki pages so they reflect the
final architecture: the benchmark/evaluation ownership boundary, the
resulting repository layout and canonical paths, capability/
progressive-loading routing, navigation and architectural terminology,
and the distinction between the first-class benchmark/evaluation
subsystem and generic documentation. The Wiki should preserve its
existing conceptual explanation of the benchmark architecture while
correcting anything — paths, ownership terminology, routing
descriptions, or navigation — that became stale because of this
relocation. The Wiki is a consumer of the architecture, not merely a
collection of links requiring mechanical path replacement; this is not
a general Wiki rewrite, and only pages actually affected by this
issue's architectural changes are in scope.
- Preserve git-history-friendly moves (e.g. `git mv`) wherever practical.
- Make progressive loading recognize benchmark/evaluation material as a
distinct class from generic documentation, so future capability
boundaries don't inherit it by accident of directory placement.
- Prevent docs-routing from becoming the accidental owner of benchmark
semantics.
- Preserve Tier 4's existing measurement/behavioral contracts exactly —
this is a location/ownership change, not a semantics change.

## Non-Goals

- No Tier 4 algorithm redesign (selection scoring, coverage math, drift
thresholds).
- No taxonomy semantics change.
- No selector redesign.
- No threshold/evidence-bar recalibration.
- No corpus content rewrite.
- No change to what a benchmark result *means* as a side effect of moving
files.
- No implementation in this issue — this issue defines scope and
scheduling only; the relocation itself is future child work opened once
both prerequisites close.
- No repository split (model.md §E.1 Model B/C already rejects this; this
issue stays inside the one-repository topology).
- No general Wiki rewrite — only benchmark/evaluation Wiki pages actually
affected by this issue's architectural changes are in scope.

## Acceptance Criteria

- [ ] An explicit ownership boundary is documented distinguishing
benchmark/evaluation infrastructure from generic documentation
- [ ] `docs/` ownership is limited to genuinely human-facing documentation
- [ ] Docs/capability routing is consistent with the new boundary
- [ ] No broken links, imports, scripts, tests, or CI references after any
move
- [ ] Benchmark execution and history behavior is unchanged unless a
change is explicitly and separately justified
- [ ] Tier 4's existing measurement/behavioral contracts are preserved
- [ ] Docs and navigation reflect the new structure
- [ ] Relevant benchmark/evaluation Wiki pages are updated to match the
final subsystem ownership, repository paths, capability routing,
and navigation, with no stale guidance that still presents
benchmark/evaluation as generic documentation or references the
pre-relocation structure
- [ ] All relevant tests are green

## Dependencies

Depends on: #412 (Tier 5 capability/progressive-loading reassessment
checkpoint — the architecture must stabilize enough, and record whether/
how the boundary pattern generalizes, before this issue's boundary can be
defined correctly), #329 (Tier 4 operational benchmark-quality loop epic —
must stabilize first; this issue must never block or be blocked by
Tier 4's active implementation work)
Blocks: none — no Tier 5 closure/integration issue exists yet to block
Parent: none — no existing issue owns this scope; #403 explicitly excludes
it (Non-Goals: "Migrating any capability besides specialist-depth", "Any
repository split"), and #412 is the checkpoint that will decide whether
follow-on capability/ownership work like this extends #403 or needs a new
epic
Related: #403, #404, #405, #406, #407, #408, #409, #410, #411 (capability-
architecture family whose manifest work this boundary would eventually
plug into), #331, #332, #333, #334, #335, #338, #339, #420 (Tier 4
benchmark-quality-loop family whose stabilization this depends on),
`docs/capability-architecture/capability-architecture-model.md` §E (repository
topology), §H (benchmark architecture)

## Validation

- Reviewed and approved by a maintainer before any child implementation
issue is opened
- Re-confirmed against #412's recorded answer once it closes

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.