Spike: RAGAS (or equivalent) eval for cited OpenCRE Chat

Task

Time-boxed research spike: stand up a faithfulness / citation eval for Chat so #1075 has a pass/fail bar beyond “it returned 5 cards”.

Preferred candidate: RAGAS (open-source RAG eval). Also consider a tiny custom harness if RAGAS’s metrics need a vendor LLM judge we cannot run in CI.

#956 is the Librarian mapping eval gate (explicit-slice 100% on the golden dataset). This spike is chat Q&A: did the answer stick to retrieved passages, and did we cite the right CRE/standard?

Success criteria (all must pass)

• 15–30 question gold set in-repo (start from #962’s threat-modeling pair, password storage, one AIX/AI-security question, one “should refuse / insufficient context”)
• Metrics named and justified: at least faithfulness (no unsourced claims) and citation/context recall (SAMM appears when the prompt says SAMM)
• Verdict: RAGAS in CI vs custom unittest over fixtures vs manual script (scripts/…, not prod)
• Kill criteria: if RAGAS requires paid judges and non-hermetic CI for the only metrics we care about, prefer a hermetic custom scorer and keep RAGAS as an optional local command
• Do not add RAGAS to requirements.txt. Optional extra in requirements-dev.txt only if the verdict is “yes, local/CI optional job”
• This spike does not merge a flaky LLM-as-judge into required make test

Context

• Chat: /rest/v1/completion, PromptHandler.generate_text
• Librarian eval (different job): scripts/evaluate_librarian.py, #956
• Related: #962, #1075, Haystack spike

Constraints

• Time-box: 1–2 days
• Mocks: gold answers can be fixtures; live Gemini optional and documented
• Production DB: N/A

Start with the Chat entry points /rest/v1/completion and PromptHandler.generate_text, then compare the existing scripts/evaluate_librarian.py flow and the related issues. Build the 15–30-question gold set and evaluate faithfulness plus citation/context recall. Done means a documented verdict on RAGAS, a hermetic custom scorer, or a manual script without adding a flaky judge to required CI.