Deterministic cross-encoder rerank between RRF fusion and the LLM judges

Local cross-encoder rerank between fusion and the LLM judges

Problem

find_similar_hybrid_scoped fuses BM25 plus dense cosine with RRF and hands the fused top N straight to the LLM judges. RRF aggregates ranks, it does not judge relevance: a memory ranking #2 in both halves on weak scores beats one ranking #1 densely with no lexical overlap. Every false positive the fusion lets through costs a judge call on the most expensive path in the memory system. There is no precision layer between cheap fusion and expensive judges.

The embedding crate already ships a CrossEncoder (ms-marco MiniLM L6 v2, 22M, tract runtime) with an if present test, and nothing calls it from memory.rs.

Correction I owe this repo

After filing I read docs/plans/MEMORY_GRAPH_PLAN.md and found this was already tried and rejected: the noisy window local CE scored recall@5 0.325 vs hybrid 0.530, because MS-MARCO is out of domain for memory statements. My first version defaulted the rerank on, which would have regressed anyone carrying the artifact. I flipped the default to off (opt in only) in SK-DEV-AI/jcode@f64cda2. The focused query variant my branch uses was never scored upstream, so it is untested rather than proven bad. Details are in the comments below.

What I built

Branch feat/rerank-fuse (fork SK-DEV-AI/jcode, now at f64cda2). Seven files, no new dependencies, no schema change.

After RRF fusion I take the top 20 candidates and rescore each query/content pair with the cross-encoder. Winners are top limit by rerank score. 20 pairs at ~6ms tract CPU is ~120ms on the recall path, not the streaming loop. Returned scores stay RRF, only the order changes, because downstream filters are calibrated on RRF magnitudes and CE logits live on an unbounded scale.

The knob is agents.memory_reranking_enabled (default false now, env override, sample config, fingerprint registry). It is artifact gated as well as config gated: the rerank only runs when ~/.jcode/models/ce-minilm-l6/ holds model.onnx plus tokenizer.json. No artifact means silent RRF only. No auto download, fetching 90MB unasked would break local first. The recall path logs one line telling the user where to put it.

Fail open everywhere: a candidate the scorer skips trails behind scored ones in RRF order, zero scored means input order passes through, load failure is sticky None with a one time warning. apply_rerank is pure so the policy tests without the artifact.

Measurement (answers the focused query ask)

The bench judge stage pins OpenAI/Claude creds I do not have, so I reproduced its exact judge contract through a local OpenAI compatible proxy with free models. 181 queries from 29 sessions, pool top 50.

recall@5: hybrid 0.000 / 0.132, hybrid_focused 0.242 / 0.199, ce_rerank_focused 0.356 / 0.349, under two independent judges. Focused CE wins under both and is the most stable across them.

I am not hiding the weak parts: the two free judges agreed exactly only 42% of the time, so absolute numbers are instrument bound and only the direction is the claim. 29 memories is a small corpus with floor effects. Our hybrid absolute does not reproduce the recorded 0.530 (different corpus and judge), so I only compare configs within my run. And this does not claim the local CE beats the shipped listwise LLM reranker (0.75). Its place is the no manager fallback, never a replacement when an LLM is available. Latency is ~120ms per recall, CPU, under 200MB.

Testing

7 new tests (policy with stub scorers, if present end to end through hybrid_fuse, config default false and env override). Two existing hybrid_fuse tests pinned to RRF with the flag off. Full jcode-base suite: 1369 passed / 10 failed on branch vs 1362 / same 10 on clean master (identical names). Zero regressions. fmt and clippy clean on touched code.

Why not bge or ModernBERT rerankers: 400MB to 2GB, need ort, 20-100ms per pair on CPU. Wrong side of the value line for a recall path. Why not return CE scores: breaks gap filter calibration. Why not auto download: unasked 90MB breaks the local first contract the embedder path respects.

Use any part of this you like, rewrite it however fits.

Start with find_similar_hybrid_scoped in memory.rs and the existing hybrid_fuse tests, then read docs/plans/MEMORY_GRAPH_PLAN.md and the branch details in the issue. Review the seven policy and configuration tests, including the default-off and environment override cases. Done means the opt-in path remains artifact-gated and fail-open, while the focused tests, fmt, clippy, and existing suite remain clean.