Resolve remaining prompt-eval defects: map/practice split, unanticipated maps, evidence retry
- Dominant language
- JavaScript
- Stars
- 26
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
## Description
PR #116 made the prompt evaluation framework trustworthy but deliberately left four known defects unfixed. Each was measured across five runs on two harnesses (2 claude, 3 codex) with the extraction prompt held at v8. None of them is a measurement problem any more, so more runs will not resolve them: each needs a decision.
1. Map and practice split (the largest remaining recall deficit).
The proposal-extract prompt files independently-actionable rules inside map bodies instead of emitting them as separate practice candidates. The usual vehicle is the evidence-gated "When changing this, verify..." clause, which fired on 14 of 14 emitted maps despite being documented as optional with an explicit warning against filling it as a template. In admit-map-vocab-01 the model emitted zero practices and merged both expected practices into one clause joined by "and". In admit-map-location-01 the whole bridge-contract-tests practice, all three facets, sits inside the clause. admit-map-feature-01 fails this way on both claude and codex in all five runs, making it the single most reproducible defect in the corpus; on codex it also surfaces as a kind mismatch where beacon-review-state-boundary matches but as a map where a practice was expected.
A clause-wording fix was authored and measured (commits 1b187a0 and ca08d59) and then REVERTED. It did not improve recall and it doubled phantom count, because forbidding the landing spot did not make the model create a separate practice node: it either dropped the content or produced a redundant map instead. The evidence supports "the content lands in the wrong place" but not "it will land in the right place if the wrong one is closed". The real target is upstream, in how the prompt decides practice versus map candidacy, not in the clause wording.
2. Unanticipated but transcript-grounded maps are scored as phantoms.
mixed-salvage-01 on claude produces a map describing two webhook retry construction paths, grounded in transcript lines 12 and 14. admit-prohibition-01 on codex produces an orchard_index queue map, in all three runs. Both pass the prompt's own independent-map-value counterfactual test, and both are charged as phantoms because the sidecar never anticipated them. Critically the affected fixture DIFFERS BY HARNESS, so enumerating the expected extras per fixture does not scale across the five shipped adapters: you cannot predict which legitimate extra map a given model will produce. Options are optional expected points excluded from the recall denominator, a small max_unexpected_proposals budget, or per-fixture auditing. The first is the only one that scales, but it weakens the contract, so this is a policy decision rather than a bug fix.
3. The judge retry does not cover judgment evidence-invalid.
The retry added in PR #116 fires only when runJudge throws during schema validation. When the judge cites evidence that is not a substring of the proposal, the judgment is schema-valid at write time and only the scorer rejects it afterwards, so no retry happens and the fixture is zeroed. Observed on admit-map-feature-01. Closing it requires the runner to validate evidence against the proposal text before accepting, which duplicates normalizeForEvidence between scripts/prompt-eval/score.mjs and src/commands/prompt-eval.ts unless that helper is shared. If the two copies drift, the runner would accept judgments the scorer rejects, so sharing the helper is preferable to duplicating it.
4. Decision-metric guidance is undocumented.
Aggregate recall varied 19 versus 16 across identical pre-fix runs, a spread larger than most prompt changes worth detecting. After the framework fixes, claude produced 18 and 18 and codex produced 19, 21 and 20. Accept or reject decisions during this work were made on per-fixture consistency across runs, not on aggregate recall, and that heuristic should be written down so the next person does not read a one-point aggregate move as signal.
## Acceptance Criteria
- [ ] The map and practice split defect is addressed at the candidacy decision rather than the change-oriented clause, and admit-map-feature-01 passes on both claude and codex across repeated runs.
- [ ] A policy is chosen and implemented for transcript-grounded maps the sidecar did not anticipate, and it does not require enumerating expected extras per fixture per harness.
- [ ] A judge response whose evidence is not present in the proposal triggers the existing retry instead of zeroing the fixture, with normalizeForEvidence shared between the scorer and the runner rather than duplicated.
- [ ] docs/internals/prompt-eval.md records that prompt changes are accepted or rejected on per-fixture consistency across runs, with the observed aggregate-recall spread cited as the reason.
- [ ] Any prompt change is validated on both claude and codex before landing, since the split defect reproduces on both.
## Additional Context
Follow-up to PR #116, which fixed the judge output schema, added a judge retry, made the evaluation hermetic, separated near misses from unexpected proposals, and repaired codex startup inside the sandbox. The extraction prompt is unchanged at v8 on that PR, so these defects are pre-existing rather than regressions.
Baseline to measure against, extraction prompt v8, hermetic runs:
claude, 2 runs: recall 18/23 and 18/23, phantom 1 and 0, near misses 0 and 2, gate 11/11 both.
codex, 3 runs: recall 19/23, 21/23 and 20/23, phantom 1 in every run, near misses 0, 0 and 1, gate 11/11 all three.
All five runs produced 24/24 valid results and 13/13 valid judgments.
Reproducing the baseline: npm run prompt-eval -- --harness claude --runs 2 --concurrency 4, and npm run prompt-eval -- --harness codex --runs 3 --concurrency 2. Use concurrency 2 for codex because higher values have previously hit provider concurrency limits. codex must be authenticated; an unauthenticated codex fails every fixture with a 401 that reproduces inside the repository too and is unrelated to the sandbox.
Contributor guide
Research direction
Start with scripts/prompt-eval/score.mjs and src/commands/prompt-eval.ts, then read docs/internals/prompt-eval.md. Reproduce the baseline with the two npm run prompt-eval commands, using the stated harness concurrency, before deciding how to address the candidacy, unexpected-map, retry, and metric-guidance issues. Done means the acceptance criteria pass across claude and codex and the documented guidance is updated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- devtools, documentation, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100