memory_search smart=true: RRF fusion collapses results to exactly 1 (SmartRetrieval unusable) — v3.10.33
- Dominant language
- TypeScript
- Stars
- 72.7k
- Forks
- 8.6k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 83
Description
## Summary
`memory_search` with `smart=true` (SmartRetrieval pipeline) no longer crashes in `v3.10.33`, but the **RRF fusion stage always collapses the result set to exactly 1**, regardless of how many candidates the vector search returns. SmartRetrieval is therefore effectively unusable: it returns 1 hit where the raw HNSW path returns the full, correctly-ranked top-k.
This is a **different bug** from the earlier `"smartSearch is not a function"` crash seen on `3.7.0-alpha.11` — that crash is fixed; the pipeline now runs end-to-end (variant expansion → RRF → recency → MMR → session), but RRF over-filters.
## Environment
- ruflo / claude-flow `3.10.33` (npm `ruflo@latest`)
- MCP server, `stdio` transport
- Windows 11
- Memory backend: `sql.js + HNSW`, 297 entries, 100% embedding coverage (Xenova all-MiniLM-L6-v2)
- Namespace under test: `claude-memories` (296 entries)
## Expected vs Actual
**Expected:** `smart=true` returns a diversified top-k (RRF should fuse and re-rank candidates, not discard them), at least comparable in recall to `smart=false`.
**Actual:** `smart=true` returns exactly 1 result on every query. The telltale is in the returned `stats`: `rawCandidateCount` is healthy (11–30) but `afterRrfCount` is always `1`. `afterRecencyCount`, `afterMmrCount`, and `afterSessionCount` then just propagate that single survivor (1 → 1 → 1). So the collapse happens **at the RRF stage**, not in MMR or recency.
## Reproduction
Same query, same namespace, only `smart` toggled.
### Query 1: `"swarm orchestration patterns"`
`smart=false` → **5 results** (similarity 0.72 → 0.47), `backend: HNSW + sql.js`
`smart=true`:
```json
{
"results": [ /* 1 item, similarity 0.44 */ ],
"total": 1,
"backend": "SmartRetrieval (RRF + MMR + Recency)",
"stats": {
"variantCount": 2,
"rawCandidateCount": 30,
"afterRrfCount": 1,
"afterRecencyCount": 1,
"afterMmrCount": 1,
"afterSessionCount": 1
}
}
```
### Query 2: `"heartbeat mail draft allowlist"`
`smart=false` → **4 results** (similarity 0.44 → 0.31)
`smart=true`:
```json
{
"results": [ /* 1 item, similarity 0.17 */ ],
"total": 1,
"backend": "SmartRetrieval (RRF + MMR + Recency)",
"stats": {
"variantCount": 2,
"rawCandidateCount": 11,
"afterRrfCount": 1,
"afterRecencyCount": 1,
"afterMmrCount": 1,
"afterSessionCount": 1
}
}
```
Both runs use `limit: 5`, so the cap is not the cause.
## Diagnosis (best guess)
The Reciprocal Rank Fusion step appears to keep only a single fused candidate. Likely causes to check: a fusion accumulator keyed so that all ranked lists collapse onto one key, an off-by-one / wrong slice on the fused output, or a default `k`/top-N of 1 applied after fusion. Secondary symptom: the surviving item's similarity score is also degraded vs the raw path (0.44 vs 0.72 for the same top hit in Query 1, 0.17 vs 0.44 in Query 2), suggesting the RRF re-scoring is feeding through as the final score.
## Impact / Workaround
- **Impact:** SmartRetrieval (query expansion + RRF + MMR diversity) is unusable in practice; any caller passing `smart=true` silently loses recall.
- **Workaround:** use `smart=false` (the default). The raw HNSW path returns the full top-k with clean similarities.
Contributor guide
Research direction
Start at the SmartRetrieval pipeline's RRF fusion stage, using the reported memory_search smart=true reproduction and its stats as the baseline. Trace how the 11–30 raw candidates become one after RRF, then verify that the fix preserves a diversified top-k and that recency, MMR, and session stages no longer receive only one result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- databases, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100