Manual eval corpus for the proposal-extraction prompt (labeled synthetic sessions, deterministic scoring, zero automation)

## Summary

Build a labeled fixture corpus of synthetic session logs plus a deterministic scorer, so that a maintainer can **manually** evaluate `proposal-extract.md` (and, through it, the `knowledge-admission.md` criteria it references) whenever a prompt `Version:` bump is considered. Today a prompt bump is validated only by the manual test plan — a checklist, not a score. This gives a bump a number: *"v6 extracted 9/10 expected knowledge points and produced 1 phantom; v5 was 7/10 with 4 phantoms."*

This is the R3 recommendation from the agentic-engineering review, with one hard constraint set by the maintainer:

> **Completely manual. No launcher pattern like `curate`. No nudges, no notifications, no hooks, no headless exec by kenkeep code, never CI.**

Kenkeep ships **zero** runtime for this. The deliverables are: committed fixtures, a dev-only scoring script, and a procedure document. A human runs every LLM step by hand.

## What is under eval

- **`src/templates-source/prompts/proposal-extract.md`** (currently `Version: 5`) — consumes a role-tagged session transcript, applies the session-disposition gate (abandoned / exploratory / unrelated / meta-only ⇒ empty output), two per-candidate filters, and emits one JSON object.
- **`src/templates-source/prompts/knowledge-admission.md`** (currently `Version: 2`) — the durability keep/drop criteria the extractor references (lifecycle actions, plan/ticket references, incidental facts, the six-months keep test, the salvage rule).

The output contract is already machine-checkable: `ProposalOutputSchema` (`{ practice: [], map: [] }` of `{ type, tags, title, description, body, kk_confidence }`, `.strict()`), registered as `proposal-output` in `src/lib/schema-registry.ts` and checkable via `node dist/cli.js validate proposal-output `.

**Out of scope for v1:** evaluating the curator phase (dedup, conflicts, modify-restraint). That needs KB-state fixtures and a second corpus; do it as a follow-up once this works.

## Execution model — every step a human action

```
1. AUTHOR (LLM, once, supervised): maintainer runs the corpus-authoring prompt below
in an ordinary interactive session → fixture files → reviews & commits them.
2. RUN (LLM, per eval, by hand): for each fixture session, the maintainer runs the
extraction prompt against it in a session THEY start themselves, and saves each
JSON output to results/.json. No kenkeep code spawns anything.
3. VALIDATE (deterministic): node dist/cli.js validate proposal-output results/.json
4. SCORE (deterministic, no LLM): node scripts/prompt-eval/score.mjs
→ per-category table + totals, exit 0 always (advisory).
5. RECORD (human): paste the score table into the PR that bumps the
prompt Version, per the existing prompt-versioning practice.
```

Explicitly **not** in scope: any CLI subcommand, any hook, any `SessionStart` nudge, any notification, any `exec` of a harness binary by kenkeep, any CI job. The scoring script lives in `scripts/prompt-eval/` (dev-only, excluded from the npm package `files`), not `src/`.

## Repository layout

```
tests/fixtures/prompt-eval/
README.md # provenance, corpus version, re-authoring & refresh procedure
sessions/
admit-convention-01.md # synthetic session log (format below)
admit-gotcha-02.md
reject-meta-only-01.md
mixed-salvage-01.md
trap-phantom-rule-01.md
...
expected/
admit-convention-01.yaml # labels for the session with the same basename
...
docs/internals/prompt-eval.md # the manual procedure, verbatim commands
scripts/prompt-eval/score.mjs # deterministic scorer (node, no deps beyond repo's)
```

**Fixture session format** — mirrors what the extraction prompt actually consumes: YAML frontmatter (`schema_version: 1`, `session_id:` a fixed, valid UUID v4, `harness: claude`, `captured_at:` a fixed ISO timestamp) and a body of role-tagged segments (`[USER]:` / `[AGENT]:`), matching the transcript-rendered session logs capture writes to `_sessions/`. Fixed timestamps/UUIDs keep fixtures deterministic.

**Expected-labels sidecar format** (`expected/.yaml`):

```yaml
fixture_id: admit-convention-01
category: admit-convention # see category list below
expect_empty: false # true for gate-reject fixtures
expected_points: # empty list when expect_empty: true
- id: cache-tags-invalidation
type: practice # practice | map
must_match_all: # lowercase substrings; ALL must appear in the
- "cache tag" # concatenated title+body of at least ONE emitted
- "custom invalidation" # proposal of the right type
must_not_match: # optional: substrings that must NOT appear in any
- "plan 12" # emitted proposal (guards salvage rewrites)
max_unexpected_proposals: 0 # phantom budget for this fixture (usually 0)
notes: >-
One sentence for the human reviewer: what this fixture is testing and why.
```

Matching is deliberately dumb — lowercase substring conjunction — because the scorer must stay deterministic and dependency-free. The corpus author compensates by choosing distinctive keywords (Drupal module names, entity types) rather than generic words.

## Scoring (deterministic)

`score.mjs` reads all sidecars + result JSONs and reports:

- **Per fixture:** PASS/FAIL with reasons (missed expected point / phantom proposal over budget / non-empty where empty expected / result file missing or schema-invalid).
- **Per category:** fixtures passed / total.
- **Aggregate:** expected-point recall (points matched / points total), phantom count (unexpected proposals across all fixtures), gate accuracy (reject fixtures that correctly produced empty output / total reject fixtures).
- Exit code **0 always** — the score is advisory to a human; nothing gates on it. Stable output ordering (sort by fixture id) so two runs diff cleanly.

## Fixture corpus design

Grounded in the **`e0ipso/kenkeep-pack-drupal`** snapshot vendored by #109 (`tests/fixtures/retrieval-eval/drupal/`) so the two evals share one realistic domain and the synthetic sessions can reference real node vocabulary (module names, entity types, conventions) instead of invented filler. The corpus author reads the pack; the fixtures must be **consistent with** its content (this matters later if the same corpus is reused for curator-phase dedup evals — sessions that teach knowledge *already in* the pack are exactly the dedup fixtures of v2).

**Categories and counts (24 sessions total):**

| Category | Count | Expected output | What it tests |
|---|---|---|---|
| `admit-convention` | 2 | 1+ practice | Human teaches a project convention |
| `admit-prohibition` | 2 | 1+ practice | "Don't do X, do Y" corrective pattern |
| `admit-gotcha` | 2 | 1+ practice | Brittle integration/race/finicky config taught after failure |
| `admit-rationale` | 1 | 1+ practice | "We use X because Y didn't handle Z" |
| `admit-tooling` | 1 | 1+ practice | Non-obvious build/test invocation taught by human |
| `admit-map-feature` | 1 | 1+ map | New system introduced; seams described |
| `admit-map-vocab` | 1 | 1+ map | Project-specific term defined |
| `admit-map-location` | 1 | 1+ map | Where a major system lives |
| `reject-abandoned` | 2 | empty | Reversal with no replacement rule |
| `reject-exploratory` | 2 | empty | Investigation, options surveyed, nothing selected |
| `reject-unrelated` | 1 | empty | Off-project general programming help |
| `reject-meta-only` | 2 | empty | Planning/scoping talk, incl. one with a rule-shaped statement |
| `reject-noise` | 2 | empty | Productive session, but only first-try code / typo fixes / exploration reads |
| `mixed-salvage` | 2 | subset | Durable rule buried in history/ticket narration — extract the rule, not the story (`must_not_match` guards) |
| `trap-phantom` | 2 | empty | The known false-positive shapes: imperative correction inside a meta-only session; incidental fact dressed as convention |

Rationale for the distribution: the confidence-bias rule says a phantom convention costs more than a missed one, so **reject/trap fixtures (11) slightly outnumber clean admits (11 + 2 mixed)** and the phantom budget is 0 almost everywhere.

## The corpus-authoring prompt (run manually, once)

The maintainer pastes this into an ordinary interactive session at the repo root (after #109 lands the pack snapshot), reviews every generated file, and commits. It is versioned here in the issue; when it runs, also commit it to `tests/fixtures/prompt-eval/README.md` for provenance.

````markdown
You are authoring an evaluation corpus for kenkeep's proposal-extraction prompt.
Work strictly inside this repository. Do not modify anything outside
`tests/fixtures/prompt-eval/`.

## Inputs to read first
1. `src/templates-source/prompts/proposal-extract.md` — the prompt under eval:
the session-disposition gate (abandoned/exploratory/unrelated/meta-only),
the scope filter, the end-state framing rule, and the confidence-bias rule.
2. `src/templates-source/prompts/knowledge-admission.md` — the keep/drop
criteria (lifecycle actions, plan/ticket references, incidental facts, the
six-months keep test, the salvage rule).
3. `PRD.md` section 6 ("What counts as knowledge") — the admit and reject lists.
4. The fixture knowledge base: every node under
`tests/fixtures/retrieval-eval/drupal/nodes/` (a vendored snapshot of the
kenkeep-pack-drupal knowledge pack). Absorb its vocabulary: module names,
entity types, named conventions, tag set.

## Your task
Generate 24 synthetic AI-coding-session logs plus one expected-labels sidecar
each, following EXACTLY the category table, counts, and sidecar YAML schema in
the corpus specification (`tests/fixtures/prompt-eval/README.md`, which contains
this prompt and the table). File layout:
- `tests/fixtures/prompt-eval/sessions/-.md`
- `tests/fixtures/prompt-eval/expected/-.yaml`

## Session log format (must match exactly)
YAML frontmatter, then role-tagged body:

---
schema_version: 1
session_id:
harness: claude
captured_at: '2026-01-15T10:00:00.000Z'
---
[USER]:
[AGENT]:
...

## Realism requirements — these decide whether the eval means anything
- Sessions are set in a fictional Drupal project that USES the fixture KB's
world: reference its real module names, entity types, and conventions so the
content is domain-plausible. Invent only what the scenario needs.
- 8–20 turns each. Include realistic agent behavior: file reads, greps, patch
summaries, test runs — the noise the extractor must see through.
- Every ADMIT fixture must contain a genuine TEACHING MOMENT: the human corrects
the agent or introduces something new that the agent could not have derived
from the codebase. The knowledge must NOT already be stated verbatim in the
fixture KB (those duplicate-teaching sessions are a different, future corpus).
- Every REJECT fixture must be a plausible, productive-looking conversation that
a naive extractor WOULD mine — that is the point. Make the traps tempting:
e.g. a meta-only planning session where the user says "let me state it as a
rule: ...", or a one-off circumstance phrased like a convention.
- MIXED-SALVAGE fixtures must interleave one durable rule with plan/ticket
narration ("this is for ticket DRP-482 ...") so that only the rewritten rule
should survive; put the ticket references in `must_not_match`.
- No real secrets, tokens, hostnames, or personal data — not even fake-looking
ones (no `sk-...`, no `password=`). The corpus is committed to a public repo.
- Vary the human's voice across sessions (terse, verbose, irritated, precise).
Do not reuse sentence templates between fixtures.

## Labeling requirements
- For each expected point choose 2–4 `must_match_all` substrings that are
DISTINCTIVE (module names, entity types, specific verbs) and appear naturally
in any faithful extraction of the rule — never generic words like "use",
"always", "config". Lowercase.
- `max_unexpected_proposals: 0` everywhere except where the category spec says
otherwise.
- Write the `notes:` field for a human reviewer in one sentence.

## Self-check loop before you finish (all deterministic, run them)
1. Frontmatter of every session parses and `session_id` values are unique,
valid UUID v4.
2. Every session has exactly one sidecar and vice versa; `fixture_id` matches
the basename.
3. For every ADMIT/MIXED sidecar: manually verify each `must_match_all` term
actually appears in the session's teaching content (else the fixture is
unwinnable). For every REJECT sidecar: `expect_empty: true` and
`expected_points: []`.
4. Category counts match the specification table exactly.
Fix any failure and re-check. Then stop and list every file you created with a
one-line description — the maintainer reviews and commits; do not commit.
````

## Manual run procedure (goes in `docs/internals/prompt-eval.md`)

1. `npm run build` (the extraction prompt template ships under `templates/`).
2. For each `sessions/.md`: open a fresh interactive session yourself, provide the extraction prompt (`src/templates-source/prompts/proposal-extract.md`) followed by the fixture's body, and save the model's final JSON object to `tmp/prompt-eval-results/.json`. (Any harness; a headless one-liner the human types themselves is also fine — the point is that *kenkeep code never launches it*.)
3. `node dist/cli.js validate proposal-output tmp/prompt-eval-results/.json` for each result.
4. `node scripts/prompt-eval/score.mjs tests/fixtures/prompt-eval tmp/prompt-eval-results`.
5. Paste the score table into the PR that bumps the prompt `Version:`; per the existing practice, the changelog already calls out prompt changes — now with numbers. Optionally append the table to `tests/fixtures/prompt-eval/RESULTS.md` to build a version-over-version history.
6. Results in `tmp/` are throwaway; only fixtures, sidecars, and (optionally) RESULTS.md are committed.

Model choice, temperature, and run count are the maintainer's call per run — record them alongside the score (LLM output varies; treat single-run scores as coarse).

## Testing (of the deterministic parts only)

- Unit-test `score.mjs` matching/reporting with a tiny inline fixture set: matched point, missed point, phantom over budget, expected-empty violated, missing result file, schema-invalid result. (A plain vitest file is fine; the script itself stays out of `src/`.)
- A repo test that walks `tests/fixtures/prompt-eval/` asserting structural invariants: session↔sidecar pairing, frontmatter parses, UUID validity, sidecar schema, category counts match the table. This keeps the corpus from rotting and runs in normal CI **without any LLM**.

## Acceptance criteria

- [ ] 24 fixture sessions + sidecars committed, reviewed by a human, matching the category table.
- [ ] `score.mjs` deterministic: same inputs ⇒ byte-identical report; exit 0 always.
- [ ] Structural-invariant test green in CI; **no CI step runs an LLM or reads results**.
- [ ] `docs/internals/prompt-eval.md` procedure written; `manual-test-plan.md` cross-references it for prompt-touching releases.
- [ ] Zero new CLI commands, hooks, nudges, notifications, or harness exec paths. `scripts/prompt-eval/` excluded from the published npm package.
- [ ] One full manual run executed against prompt v5 and its score recorded as the baseline (in RESULTS.md or the landing PR description).

## Non-goals

- No launcher/`curate`-style automation, no headless exec by kenkeep, no CI LLM runs, no notifications (maintainer constraint).
- No curator-phase eval (dedup/conflict/modify) — future corpus; the pack-consistency rule above keeps the door open.
- No LLM-as-judge scoring — matching is substring-deterministic; the human reads the report.
- No score gate anywhere — advisory only.

## Dependencies / related

- **Depends on #109** for the vendored `kenkeep-pack-drupal` snapshot (the corpus grounds its vocabulary in it). If #109 stalls, the corpus can proceed with invented-but-consistent Drupal vocabulary at the cost of the shared-domain benefit — flag before choosing that path.
- Prompt versioning practice: `practice-bump-prompt-version-comment`. Schema/validate loop: `src/lib/schema-registry.ts` (`proposal-output`).
- R3 in the agentic-engineering review; #112 (R2) and PR #111 are the siblings from the same review.

Read src/templates-source/prompts/proposal-extract.md and knowledge-admission.md, then inspect the Drupal fixtures under tests/fixtures/retrieval-eval/drupal/nodes/. Add the specified sessions, expected YAML sidecars, docs/internals/prompt-eval.md, and scripts/prompt-eval/score.mjs. Build the project and validate sample results with node dist/cli.js validate proposal-output; done means deterministic per-fixture and aggregate scoring with stable ordering, always exiting 0, plus the documented manual procedure.

Written by the indexing model from the issue text.