Add faithfulness and coverage evaluation for firewall summaries and extracted facts

Summary

Add an evaluation harness that measures whether firewall summaries and extracted facts preserve the load-bearing information of the original tool output — deterministic coverage metrics by default, with an optional model-judged faithfulness check behind the existing call_fn pattern.

Why this matters

The firewall's value proposition is "smaller context, same task success." Token-savings metrics show the smaller; nothing currently measures the same. A summary that drops the error code, the row count, or the order ID silently degrades downstream answers. Coverage metrics make summarizer changes (rule edits, prompt changes, the planned #384 expansion) safe to evolve, and complement the end-to-end answer-quality harness (#445) at the component level where regressions are cheapest to localize.

Current evidence

• src/contextweaver/eval/context.py measures budget utilization and token savings — no fidelity dimension.
• src/contextweaver/summarize/rules.py (~lines 94–182): keyword-driven key-line extraction (error/success/failed/count/status) with no measurement of what is lost.
• FirewallStats (src/contextweaver/envelope.py) records chars/tokens before and after, but not information retention.
• benchmarks/e2e_quality.py checks final answer substrings with a stub model (6 tasks) — there is no component-level fidelity metric for the firewall itself.

Proposed implementation

1. Build a golden corpus (benchmarks/firewall_gold/) of representative tool outputs — API JSON, error payloads with stack traces, paginated lists, log excerpts, CSV-ish text — each annotated with its key facts (typed: identifiers, quantities, statuses, error names).
2. Deterministic metrics in contextweaver.eval (e.g. evaluate_firewall_fidelity):
   • Key-fact coverage: fraction of annotated facts findable (normalized string/number match) in the summary + extracted facts + projected fields.
   • Numeric integrity: all annotated quantities reproduced exactly.
   • Error-state retention: error/status facts retained when present.
   • Compression context: report coverage at the achieved compaction ratio so quality/size trade-offs are visible.
3. Optional model-judged faithfulness (off by default, opt-in via env like the existing CW_SMOKE_LLM pattern): user-supplied call_fn scores summary faithfulness; never run in CI.
4. make firewall-fidelity target; track per-strategy results (rule-based vs StructuredFirewall vs LLM-backed) in the scorecard.

Example prompt, schema, or interface

{
  "case_id": "bigquery-rowset-16kb",
  "key_facts": [
    {"kind": "quantity", "name": "row_count", "value": "1842"},
    {"kind": "identifier", "name": "job_id", "value": "bq-7731"},
    {"kind": "status", "name": "state", "value": "DONE"}
  ],
  "expect_coverage_at_least": 1.0
}

Acceptance criteria

• A committed, documented golden corpus of at least 30 annotated outputs across at least 5 content shapes.
• Deterministic fidelity report, byte-stable across runs, per firewall strategy.
• Coverage regressions in summarizer/extractor changes are visible in the benchmark delta comment.
• Optional LLM-judge path skips cleanly when not configured.

Test and evaluation plan

• Unit tests for the matching/normalization logic (number formats, case, unit suffixes).
• Baseline run committed; weekly scorecard tracks drift.
• Cross-check: cases where StructuredFirewall with a correct allow-list scores 100% coverage validate the metric itself.

Migration notes

Not expected to require migration.

Risks and tradeoffs

String-match coverage under-credits legitimate paraphrase — acceptable at the component level where summaries should preserve literals. Annotating the corpus is manual effort; start small and grow with regressions found in the wild.

Suggested labels

evals, ai, reliability, area/context

Start by reading src/contextweaver/eval/context.py, src/contextweaver/summarize/rules.py, src/contextweaver/envelope.py, and benchmarks/e2e_quality.py to understand existing metrics and firewall strategies. Then review the proposed benchmarks/firewall_gold/ corpus format and the call_fn and CW_SMOKE_LLM patterns. Done means a documented 30-case corpus, deterministic per-strategy fidelity reports with tests, a make firewall-fidelity target, and a cleanly skipped optional judge.