Benchmark Ask Mantosh retrieval against live production data

Problem

The repository has deterministic offline evaluation, but docs/SYSTEM_STATE.md notes that it does not measure live Vectorize recall or live-production retrieval behavior.

Goal

Add a bounded production retrieval benchmark that measures the real deployed retrieval path without storing visitor questions or consuming uncontrolled quota.

Scope

• Define a reviewed benchmark set derived from public evidence only.
• Exercise the deployed retrieval stack on a controlled schedule or explicit release check.
• Measure retrieval success, source correctness, latency, and failure rate.
• Keep benchmark traffic distinguishable from visitor traffic without logging private data.
• Set explicit pass/fail thresholds before the benchmark gates releases.

Acceptance criteria

• Benchmark cases are version-controlled and evidence-backed.
• The benchmark runs against the deployed production retrieval path.
• Results include retrieval/source correctness and latency.
• Benchmark traffic does not pollute visitor analytics or expose private inputs.
• Documentation explains when a regression should block deployment.

Start by reading docs/SYSTEM_STATE.md to understand the existing offline evaluation and the gap in live retrieval measurement. Then identify the deployed retrieval entry point and define the version-controlled, public-evidence benchmark cases, metrics, privacy boundaries, schedule, and release thresholds. Done means the production path is exercised with distinguishable traffic and documented pass/fail behavior without storing private inputs.