evaluation(context): assess Jev for semantic relevance and evidence coverage

Tracking issue: #1649

Feature description

Evaluate whether Jev can identify semantically irrelevant, incomplete, or contradictory context that passes PowerContext's deterministic retrieval checks. Begin with offline replay and advisory observations; use the evidence to decide whether an opt-in runtime integration is justified.

Priority: P1 — semantic assessment beyond candidate ranking.

Problem and proposed solution

A context pack can contain several high-scoring entries while still omitting the fact needed to answer a question. It can also contain a relevant entry that contradicts the question's premise. Candidate count and lexical overlap do not distinguish these cases.

Assess bounded, authorized candidates against explicit task requirements. Ask narrow questions about each item's direct usefulness, evidence support, and contradictions, then assess coverage of supplied requirements. Return an advisory result such as supported, partial, conflicting, or unknown, with exact candidate references and the model's probabilities. Missing information must remain unknown; the model cannot establish that the full repository contains no evidence by inspecting a shortlist.

The evaluator should separately inspect what survived final byte-budget fitting: adequate retrieval does not imply that the delivered context still contains the required evidence. Preserve contradictory evidence for inspection instead of silently discarding it. Date ordering, revision validity, permissions, and byte counts remain deterministic checks.

Acceptance criteria

• A replay evaluator consumes explicit task requirements, exact candidate references, and final delivered selections without changing production selection or persistence.
• Compare the existing model-free gate alone against its signals plus Jev assessment, using the same retrieval inputs and output budget.
• Measure false-sufficient decisions, false-insufficient decisions, contradiction recall, useful-evidence retention, and added latency/cost; include bilingual, adversarial, and budget-truncation cases.
• Evaluate both individual claims and requirements needing complementary evidence. Pin model/prompt versions and calibrate thresholds on data separate from the held-out evaluation.
• Low confidence, unavailable evidence, and provider failures produce an explicit unknown/fallback observation and preserve normal prepare behavior.
• Propose runtime integration only if the measured benefit justifies the added request-path cost. Any expansion policy remains bounded and cannot widen authorized Scopes, caller-selected families, or the output budget.
• Keep PreparedContext and its trust envelope unchanged. Any new public diagnostic contract is designed through the existing RFC process and coordinated with #1356.

Alternatives considered

The existing deterministic gate is the default baseline and may be sufficient. A stronger LLM judge or task-specific deterministic evidence checks may outperform Jev. The experiment should support a decision not to add another runtime model call.

Additional context

#1596 has implemented the model-free gate proposed by #1556. This work evaluates semantic coverage and evidence relationships, not that gate's implementation. The Jev Memory reranker orders candidates within one family; this experiment asks whether the selected evidence actually supports the supplied requirements. It is also separate from representation/progressive retrieval in #1426 and Artifact evolution in #1635.

• Current deterministic gate
• TypeSafe RAG passage classification
• Known model limitations
• Confidence semantics

Prepared with AI assistance. This is a proposed experiment, not evidence that a model-based gate improves PowerContext.

Related Jev work: #1643 ranks Memory candidates; this issue evaluates semantic coverage. Reuse provider/evaluation utilities while calibrating this task separately.