dragonflydb / dragonflydb/dragonfly
Add remaining scorers (BM25STD.NORM, BM25STD.TANH, DISMAX, DOCSCORE)
- Dominant language
- C++
- Stars
- 31.5k
- Forks
- 1.3k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 137
Description
Dragonfly search currently supports three scorers: BM25STD (default), TFIDF and TFIDF.DOCNORM are in progress. For full feature parity with competitors, the following scorers still need to be implemented.
Scorers to add:
**BM25STD.NORM**
Min-max normalized BM25STD. Scores are rescaled to [0, 1] across the result set:
`normalized_score = (score - min_score) / (max_score - min_score)`
Requires a two-pass approach in `TakeScoredTopK`:
1. First pass: compute raw BM25STD scores for all matched docs, track min/max
2. Second pass: rescale before returning top-K
Alternatively, compute raw scores, then normalize just the top-K before output (simpler but min/max are over the top-K subset, not the full result set - matches Redis Stack's behavior).
See: https://redis.io/docs/latest/develop/ai/search-and-query/advanced-concepts/scoring/
**BM25STD.TANH**
BM25STD with tanh-based normalization: `tanh(score * alpha)` where alpha is a configurable sharpness parameter. Scores are bounded in (0, 1) without requiring min/max. Single-pass, but requires an API way to pass the alpha parameter.
**DISMAX**
Disjunction Max: takes the **maximum** per-term score instead of the sum. Changes the aggregation in `ScoreDocument` from `+=` to `std::max`. Requires either a new path in the switch or a template parameter/callback for the aggregation operator.
Useful for OR-heavy queries where summing multiple term scores over-inflates documents that match many loosely-related terms.
**DOCSCORE**
Returns the document's static score set at index time (currently `DocIndex.default_score = 1`). Does not use term frequencies at all.
Requires:
- Per-document score storage (if FT.CREATE ever gains a per-doc WEIGHT parameter)
- Or just returns the index-wide `default_score` for all matches (simpler, matches current Redis behavior when no per-doc score is set)
- Plumbing `default_score` from `DocIndex` through to the scoring context
**HAMMING**
Hamming distance between query payload and document payload. Dragonfly does not support document payloads, so this scorer is out of scope until/unless payload support is added.
- Scoring documentation: https://redis.io/docs/latest/develop/ai/search-and-query/advanced-concepts/scoring/
- Current implementation: #7093 (TF storage), #7101 (BM25STD engine), #7181 (API integration)
Contributor guide
Research direction
Start by reading the current scoring work referenced in #7093, #7101, and #7181, then inspect TakeScoredTopK, ScoreDocument, and DocIndex.default_score. Define the scoring and API decisions for BM25STD.NORM, BM25STD.TANH, DISMAX, and DOCSCORE, and verify that all four scorers return the intended results; HAMMING remains out of scope without document payloads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100