Chat: retrieve top-k via Librarian seams (implements #962)

Task

Stop OpenCRE Chat (POST /rest/v1/completion) from stuffing one nearest embedding into the prompt. Reuse the Librarian retrieve + rerank seams so answers are grounded in several passages and the existing references UI can show more than one card.

This implements #962. GSoC Module C is in-tree; the old “blocked on cheat-sheet ETL” gate on #962 is no longer the blocker. Shared retrieval should also be the backend #1074 wraps later — do not invent a second RAG stack.

Why not LibrarianPipeline as-is

LibrarianPipeline is C.0→C.4 mapping (chunk → CRE link / review). Chat is Q&A. Reuse components, not the queue runner:

• application/utils/librarian/factory.py build_components() / build_retriever()
• application/utils/librarian/candidate_retriever.py (pgvector top-k)
• application/utils/librarian/title_boost.py (lexical rerank, no torch)
• application/utils/librarian/cross_encoder.py CrossEncoderReranker only where sentence-transformers is installed

Chat today searches Standard nodes (get_id_of_most_similar_node_paginated). C.1’s default pool is the CRE hub. The chat path needs both: top-k standards (for SAMM / cheat sheets / AIX) and top-k CREs, then linked neighbors.

Success criteria (all must pass)

• PromptHandler.generate_text retrieves top-k (default 8 retrieve / 5 in prompt; env-tunable, do not hardcode a second threshold stack if CRE_LIBRARIAN_TOP_K_* already fits)
• Prompt instructs the model to answer only from those passages and to cite them; no silent query_llm essay when nothing is above threshold (accurate=false + existing warning is OK)
• Response table lists all passages used (CRE id and/or standard section + embeddingsUrl). Frontend chatbot.tsx already maps data.table as reference cards — do not add a second citation UI unless the payload is missing CRE ids
• Heroku slug: do not add sentence-transformers / torch to requirements.txt. Prod chat must work with pgvector top-k + title_boost (and optional API rerank later). Local/dev may enable MiniLM CE when requirements-dev.txt is installed
• Tests: generate_text with a stub retriever returns multiple table rows and a prompt containing more than one passage; below-threshold path does not claim accurate=true. Extend application/tests/chat_completion_test.py and add a focused prompt_client test
• make lint / make mypy / targeted unittest green
• Manual: Rob’s #962 pair — “how to pick the right people to join in threat modeling” vs the same according to SAMM — SAMM must appear in references for the second; the first must not be a single cheat-sheet blob if other hits exist

Context

• Chat: application/prompt_client/prompt_client.py generate_text (argmax-1, truncate 8000 chars)
• Route: application/web/web_main.py chat_cre → /rest/v1/completion
• UI: application/frontend/src/pages/chatbot/chatbot.tsx (data.map reference cards)
• Librarian: application/utils/librarian/README.md, factory.py, candidate_retriever.py
• Related: #962 (product), #965 (MyOpenCRE reuse same seams), #1074 (NestBot cited ask — consume this layer), #956 (librarian eval CI)

Constraints

• In scope: chat retrieval + prompt + table payload
• Out of scope: NestBot machine auth (#1074), MCP ask, replacing chat with Onyx/Haystack/Dify, audio overviews, unbounded web search
• Dependencies: none new. No LlamaIndex on the prod slug
• Mocks: stub retriever/rerank in unit tests; do not mock away the multi-hit contract
• Production DB: N/A (read embeddings only)

Suggested shape (not a second pipeline)

1. Embed the user question (existing get_text_embeddings).
2. pgvector top-k Standard + top-k CRE (extend find_most_similar_embedding_id or instantiate PgVectorRetriever with doc_type).
3. Title-boost / hybrid lexical rerank (title_boost.py). CE if importable, else skip.
4. Load embeddings_content + hyperlink for the shortlist; cap total prompt chars.
5. create_chat_completion with the concatenated passages; return all rows in table.

Start with application/prompt_client/prompt_client.py and the existing generate_text path, then read application/utils/librarian/factory.py, candidate_retriever.py, and title_boost.py. Extend the focused tests in application/tests/chat_completion_test.py and add the requested prompt_client test using stub retrieval and reranking. Done means multiple passages appear in the prompt and table, threshold failures are not marked accurate, and lint, mypy, and targeted tests pass.