afx-team / afx-team/hebb-mind

feat(retrieval): make the strict-recall score floor configurable + recalibrate for the cross-encoder scale

Đang mở
#31 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
area: usability effort: simple retrieval
Ngôn ngữ chính
Python
Star
52
Fork
18
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

The two production recall surfaces — the MCP server and the Claude Code recall hook — opt into a strict relevance floor so an agent never gets back loosely-relevant noise. The usability risk (好用 / retrieval correctness) is that a floor calibrated on the wrong score distribution can silently return an **empty** set on legitimate hits. Audit C5 / 召回F3 flagged that a single `recall_min_score=0.8` applied across the post-rerank mixed scale (cross-encoder sigmoid for the reranked pool vs. composite `disp_score` for the tail) drops correct-but-non-literal short-conversation hits, consistent with our own MemBench Hit@1 ≈ 0.49.

**Most of the original proposal already landed in PR #24** — this issue narrows to what remains: validating the recalibration against a retrieval dataset and finishing the per-call knob on the two surfaces.

## Current state (verified against the tree)

Already shipped (PR #24 — narrowing, not re-doing):
- `recall_min_score` is a config setting, default `0.8`, console-editable — `src/hebb/config/settings.py:108-113`.
- `MemoryQuery` exposes a per-call `min_score` (0–1, 0 = no filter) and a `strict_recall` opt-in flag — `src/hebb/models/memory.py:87-93`.
- The router applies the configured floor only when `strict_recall` is set and the caller did not pass an explicit `min_score` — `src/hebb/server/routers/search.py:39-40`.
- The dual-scale problem is addressed: the floor is translated to the rerank scale for reranked entries and kept on the composite scale for the tail, via `_RERANK_FLOOR_RATIO = 0.625` — `src/hebb/retrieval/searcher.py:40-48` and the floor loop at `src/hebb/retrieval/searcher.py:289-296`.
- Unit coverage for the dual-scale floor exists — `tests/unit/test_audit_retrieval.py:164-185` (`test_below_rerank_floor_hit_is_dropped`, `test_no_rerank_floor_uses_composite_scale`) and the "0.6 sigmoid survives a 0.8 composite floor" case at `tests/unit/test_audit_retrieval.py:150-161`.

What still remains:
- **The `0.625` ratio and the `0.8` default are heuristics, not eval-validated.** The comment derives `0.625` from "a relevant cross-encoder hit clears ~0.5 sigmoid" (`src/hebb/retrieval/searcher.py:46-47`); there is no recall@k measurement that the shipped default does **not** silently empty strict recall on a dataset. The audit's own remediation item C5 explicitly asks to "为 bge sigmoid 重定门槛" — the threshold was relaxed but never measured.
- **The two production surfaces still cannot pass a per-call floor.** Both hardcode `strict_recall: True` and send no `min_score`: MCP at `src/hebb/mcp/server.py:111`, hook at `src/hebb/integrations/claude_code/recall.py:138`. The per-call `min_score` plumbing exists in the model/router but no caller threads a configurable value through these two surfaces, so the floor is still effectively un-overridable at the call site.
- No eval probe reports the non-empty-recall rate at the default floor; existing probes (`eval/kw_probe.py`, `eval/retrieval_lab.py`) don't exercise `min_score`.

## Proposed approach

- Add an offline probe (extend `eval/kw_probe.py` or `eval/retrieval_lab.py`) that, on at least one labelled retrieval dataset, reports recall@k **with the strict floor applied** vs. unfiltered, plus the fraction of queries the floor empties. Use it to confirm/retune `recall_min_score` (default `0.8`) and `_RERANK_FLOOR_RATIO` (`0.625`) so the shipped default does not silently empty recall.
- Promote `_RERANK_FLOOR_RATIO` from a module constant to a `Settings` field (console-editable) so the rerank-scale mapping is tunable without a code change, mirroring `recall_min_score`.
- Let the MCP server and the recall hook thread a configurable `min_score` (defaulting to the config floor) instead of only sending the `strict_recall` boolean, so per-surface overrides are possible.

## Acceptance criteria

- [ ] An eval probe reports, for a labelled dataset, recall@k under the strict floor and the fraction of queries returning an empty set; results are committed to the run report.
- [ ] The shipped default floor is confirmed (or retuned) so it does **not** silently empty strict recall on that eval set.
- [ ] `_RERANK_FLOOR_RATIO` is exposed as a config setting alongside `recall_min_score`.
- [ ] MCP (`src/hebb/mcp/server.py`) and the recall hook (`src/hebb/integrations/claude_code/recall.py`) can pass a per-call `min_score`, defaulting to the configured floor.
- [ ] Any change that **reorders** results is A/B-tested on a retrieval dataset (recall@k) before shipping; monotonic floor-only changes that merely admit/drop already-ranked results may ship without an A/B.

## Scope / out of scope

In scope: the strict-recall floor, its rerank-scale translation, and threading a per-call floor through the two production surfaces. Out of scope: the broader rerank/RRF ranking pipeline, the IDF "dead code" calibration (audit 召回F2), and embedder unit-normalization (audit C5) — those are separate items.

## References

- `reports/audit/core-system-audit-2026-06-07.md` — C5 / 召回F3 (lines ~90-97, 202): the dual-scale 0.8-floor mismatch and the "为 bge sigmoid 重定门槛" remediation.
- `reports/audit/newuser-experience-audit-2026-06-08.md` — new-user recall expectations.
- `reports/design/capability-gap-roadmap-2026-06-11.md` — roadmap row **S8** (make the strict-recall threshold configurable + recalibrate for the bge sigmoid).

Filed from the capability-gap roadmap (reports/design/capability-gap-roadmap-2026-06-11.md).

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.