feat(retrieval): make the strict-recall score floor configurable + recalibrate for the cross-encoder scale
- Ngôn ngữ chính
- Python
- Star
- 52
- Fork
- 18
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
The two production recall surfaces — the MCP server and the Claude Code recall hook — opt into a strict relevance floor so an agent never gets back loosely-relevant noise. The usability risk (好用 / retrieval correctness) is that a floor calibrated on the wrong score distribution can silently return an **empty** set on legitimate hits. Audit C5 / 召回F3 flagged that a single `recall_min_score=0.8` applied across the post-rerank mixed scale (cross-encoder sigmoid for the reranked pool vs. composite `disp_score` for the tail) drops correct-but-non-literal short-conversation hits, consistent with our own MemBench Hit@1 ≈ 0.49.
**Most of the original proposal already landed in PR #24** — this issue narrows to what remains: validating the recalibration against a retrieval dataset and finishing the per-call knob on the two surfaces.
## Current state (verified against the tree)
Already shipped (PR #24 — narrowing, not re-doing):
- `recall_min_score` is a config setting, default `0.8`, console-editable — `src/hebb/config/settings.py:108-113`.
- `MemoryQuery` exposes a per-call `min_score` (0–1, 0 = no filter) and a `strict_recall` opt-in flag — `src/hebb/models/memory.py:87-93`.
- The router applies the configured floor only when `strict_recall` is set and the caller did not pass an explicit `min_score` — `src/hebb/server/routers/search.py:39-40`.
- The dual-scale problem is addressed: the floor is translated to the rerank scale for reranked entries and kept on the composite scale for the tail, via `_RERANK_FLOOR_RATIO = 0.625` — `src/hebb/retrieval/searcher.py:40-48` and the floor loop at `src/hebb/retrieval/searcher.py:289-296`.
- Unit coverage for the dual-scale floor exists — `tests/unit/test_audit_retrieval.py:164-185` (`test_below_rerank_floor_hit_is_dropped`, `test_no_rerank_floor_uses_composite_scale`) and the "0.6 sigmoid survives a 0.8 composite floor" case at `tests/unit/test_audit_retrieval.py:150-161`.
What still remains:
- **The `0.625` ratio and the `0.8` default are heuristics, not eval-validated.** The comment derives `0.625` from "a relevant cross-encoder hit clears ~0.5 sigmoid" (`src/hebb/retrieval/searcher.py:46-47`); there is no recall@k measurement that the shipped default does **not** silently empty strict recall on a dataset. The audit's own remediation item C5 explicitly asks to "为 bge sigmoid 重定门槛" — the threshold was relaxed but never measured.
- **The two production surfaces still cannot pass a per-call floor.** Both hardcode `strict_recall: True` and send no `min_score`: MCP at `src/hebb/mcp/server.py:111`, hook at `src/hebb/integrations/claude_code/recall.py:138`. The per-call `min_score` plumbing exists in the model/router but no caller threads a configurable value through these two surfaces, so the floor is still effectively un-overridable at the call site.
- No eval probe reports the non-empty-recall rate at the default floor; existing probes (`eval/kw_probe.py`, `eval/retrieval_lab.py`) don't exercise `min_score`.
## Proposed approach
- Add an offline probe (extend `eval/kw_probe.py` or `eval/retrieval_lab.py`) that, on at least one labelled retrieval dataset, reports recall@k **with the strict floor applied** vs. unfiltered, plus the fraction of queries the floor empties. Use it to confirm/retune `recall_min_score` (default `0.8`) and `_RERANK_FLOOR_RATIO` (`0.625`) so the shipped default does not silently empty recall.
- Promote `_RERANK_FLOOR_RATIO` from a module constant to a `Settings` field (console-editable) so the rerank-scale mapping is tunable without a code change, mirroring `recall_min_score`.
- Let the MCP server and the recall hook thread a configurable `min_score` (defaulting to the config floor) instead of only sending the `strict_recall` boolean, so per-surface overrides are possible.
## Acceptance criteria
- [ ] An eval probe reports, for a labelled dataset, recall@k under the strict floor and the fraction of queries returning an empty set; results are committed to the run report.
- [ ] The shipped default floor is confirmed (or retuned) so it does **not** silently empty strict recall on that eval set.
- [ ] `_RERANK_FLOOR_RATIO` is exposed as a config setting alongside `recall_min_score`.
- [ ] MCP (`src/hebb/mcp/server.py`) and the recall hook (`src/hebb/integrations/claude_code/recall.py`) can pass a per-call `min_score`, defaulting to the configured floor.
- [ ] Any change that **reorders** results is A/B-tested on a retrieval dataset (recall@k) before shipping; monotonic floor-only changes that merely admit/drop already-ranked results may ship without an A/B.
## Scope / out of scope
In scope: the strict-recall floor, its rerank-scale translation, and threading a per-call floor through the two production surfaces. Out of scope: the broader rerank/RRF ranking pipeline, the IDF "dead code" calibration (audit 召回F2), and embedder unit-normalization (audit C5) — those are separate items.
## References
- `reports/audit/core-system-audit-2026-06-07.md` — C5 / 召回F3 (lines ~90-97, 202): the dual-scale 0.8-floor mismatch and the "为 bge sigmoid 重定门槛" remediation.
- `reports/audit/newuser-experience-audit-2026-06-08.md` — new-user recall expectations.
- `reports/design/capability-gap-roadmap-2026-06-11.md` — roadmap row **S8** (make the strict-recall threshold configurable + recalibrate for the bge sigmoid).
Filed from the capability-gap roadmap (reports/design/capability-gap-roadmap-2026-06-11.md).
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.