[Parked pending #445/#489 findings] Context-sensitivity diagnostic probes

Context

A useful framing from a recent interpretability discussion: context does not merely add information; it can bend the model's decision path toward a correct or incorrect interpretation.

For contextweaver, the practical opportunity is black-box and product-facing: given a compiled context pack, can we detect when small context changes flip routing, tool choice, or answer interpretation?

This complements, but is not the same as:

• #719 — substrate-aware context selection to reduce interpretation drift.
• #499 — context contracts validated at build time.
• #767 — lifecycle-aware LessonWeaver context sources.
• dgenio/weaver-spec#182 — proposed shared ContextDiagnostic contracts.

Problem

A context pack can be token-efficient and policy-compliant but still harmful if it pulls the model into the wrong basin:

• a stale item makes the model prefer an old workflow;
• a nearby concept causes entity confusion;
• an optional note dominates the actual task;
• removing one item flips the selected tool or route;
• adding a lesson/rule changes behavior outside its intended scope.

Today, these failures are usually found by anecdote after an agent behaves strangely. We should make them inspectable before adoption or deployment.

Proposal

Add an experimental diagnostic path that can run minimal-pair probes over context packs.

Possible CLI shape:

contextweaver probe context-pack path/to/pack.json --target route --perturb remove-one --json

Possible probe modes:

1. Remove-one — remove each context item and compare route/tool/answer class.
2. Add-distractor — inject controlled distractors and measure whether the result changes.
3. Stale-vs-current — compare old/current variants of the same fact.
4. Scope boundary — test whether a rule/lesson affects tasks outside its declared scope.
5. Budget boundary — compare packed context at different budgets and record instability.

Output

Emit a machine-readable diagnostic report, ideally aligned with dgenio/weaver-spec#182 if that contract lands:

• baseline pack ID / hash;
• task phase and target being evaluated;
• perturbation applied;
• observed route/tool/answer class before and after;
• instability score / flip indicator;
• candidate context attractor(s);
• provenance for the context items involved;
• reproducibility metadata.

Non-goals

• No white-box transformer internals.
• No model-in-the-loop routing in the normal request path.
• No automatic removal of context items without an explicit operator decision.
• No claim that the metric proves semantic correctness; it is a diagnostic surface.

Acceptance criteria

• Add an experimental probe API or CLI for at least one perturbation mode.
• Provide a minimal deterministic fixture where a context perturbation changes a routing or answer-class decision.
• Emit a JSON report with baseline, perturbation, observed delta, and candidate cause.
• Document how this differs from normal evals, context contracts, and budget checks.
• Keep the normal context build path deterministic and unchanged unless probes are explicitly invoked.

Start from the proposed contextweaver probe context-pack CLI shape and the acceptance criteria. Choose one perturbation mode and define a deterministic fixture showing a routing or answer-class change, then specify the JSON report fields and documentation distinctions. Confirm that normal context building remains unchanged unless probes are explicitly invoked.