[Parked pending evidence need] Mutation-testing pilot

Summary

Run a scoped mutation-testing pilot (e.g., with mutmut or cosmic-ray) on two or three core modules — candidate selection/scoring and the beam-search navigator — to measure how much of the 1,800+-test suite's coverage is assertion-backed, and decide whether mutation testing earns a permanent place in the toolchain.

Priority: P3 · Confidence: Low (exploratory)

Why this matters

Line coverage says code was executed; mutation score says bugs would be caught. For a library whose value proposition is deterministic correctness (budgets enforced, drops attributed, ties broken stably), surviving mutants in selection.py or navigator.py would identify exactly where regressions could slip through today's gates. A bounded pilot produces evidence before committing to ongoing cost.

Current evidence

• ~1,848 tests across 66 files with --cov in CI, but no coverage threshold and no mutation tooling anywhere in the repo (verified: no mutmut/cosmic-ray references).
• High-leverage targets with dense logic and clear invariants: src/contextweaver/context/selection.py, context/scoring.py, context/dedup.py, routing/navigator.py.
• Adjacent open issues: #475 (coverage ratchet) measures execution coverage; #440 (property-based tests) generates inputs. Mutation testing evaluates the assertions — a third, distinct axis.

External context

mutmut is the most common Python mutation tool; typical practice is per-module runs with a baseline cache, not whole-repo gating, because runtime grows with suite size × mutant count.

Proposed implementation

1. Pick 2–3 modules (recommend selection.py, dedup.py, navigator.py); run mutation testing against the focused test files (-k scoping) to keep runtime sane.
2. Triage surviving mutants into: (a) missing assertions → write tests; (b) equivalent mutants → document; (c) dead/unreachable code → candidate cleanup.
3. Write up results in the issue: mutation score per module, tests added, and a recommendation (adopt as scheduled non-gating job / adopt for security-grade modules only / do not adopt).
4. If adopted: add a make mutation target and a weekly scheduled workflow, never a PR gate.

AI-agent execution notes

• Inspect first: the target modules and their test files (tests/test_selection.py, test_dedup.py, test_navigator.py); pyproject.toml test config.
• Run: pip install mutmut, scope runs per module; expect long runtimes — use --paths-to-mutate and targeted test selection.
• Edge cases: mutmut + pytest-asyncio interplay; cache files must be gitignored.
• Preserve: do not change source to please the tool; only add tests or report findings.
• Do not wire anything gating into make ci in this pilot.

Acceptance criteria

• Mutation scores reported for the pilot modules with surviving-mutant analysis.
• At least the clearly-missing assertions found (if any) are converted into merged tests.
• A documented adopt/don't-adopt recommendation with runtime costs.

Test plan

The pilot is the test plan; any new tests it produces run in the normal suite.

Documentation plan

Findings summarized in this issue; if adopted, a section in docs/agent-context/workflows.md; CHANGELOG only if tooling lands.

Migration and compatibility notes

Not expected to require migration.

Risks and tradeoffs

Mutation runs are slow and can produce noisy equivalent mutants (time-boxed pilot mitigates); risk of over-fitting tests to mutants (review for meaningfulness). The cheap outcome — "our assertions are strong, no action" — is itself valuable evidence.

Suggested labels

testing, investigation

Inspect src/contextweaver/context/selection.py, context/scoring.py, context/dedup.py, routing/navigator.py, their focused tests, and pyproject.toml. Run a scoped mutmut pilot on two or three modules with targeted tests, then analyze surviving mutants. Done means reporting mutation scores, converting clearly missing assertions into tests where appropriate, and documenting runtime costs plus an adopt-or-decline recommendation.