dragonflydb / dragonflydb/dragonfly

Add remaining scorers (BM25STD.NORM, BM25STD.TANH, DISMAX, DOCSCORE)

Open
#7,199 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
31.5k
Forks
1.3k
Avg merge
1d 10h
Merged PRs (30d)
137

Description

Dragonfly search currently supports three scorers: BM25STD (default), TFIDF and TFIDF.DOCNORM are in progress. For full feature parity with competitors, the following scorers still need to be implemented.

Scorers to add:

**BM25STD.NORM**
Min-max normalized BM25STD. Scores are rescaled to [0, 1] across the result set:
`normalized_score = (score - min_score) / (max_score - min_score)`

Requires a two-pass approach in `TakeScoredTopK`:
1. First pass: compute raw BM25STD scores for all matched docs, track min/max
2. Second pass: rescale before returning top-K

Alternatively, compute raw scores, then normalize just the top-K before output (simpler but min/max are over the top-K subset, not the full result set - matches Redis Stack's behavior).

See: https://redis.io/docs/latest/develop/ai/search-and-query/advanced-concepts/scoring/

**BM25STD.TANH**
BM25STD with tanh-based normalization: `tanh(score * alpha)` where alpha is a configurable sharpness parameter. Scores are bounded in (0, 1) without requiring min/max. Single-pass, but requires an API way to pass the alpha parameter.

**DISMAX**
Disjunction Max: takes the **maximum** per-term score instead of the sum. Changes the aggregation in `ScoreDocument` from `+=` to `std::max`. Requires either a new path in the switch or a template parameter/callback for the aggregation operator.

Useful for OR-heavy queries where summing multiple term scores over-inflates documents that match many loosely-related terms.

**DOCSCORE**
Returns the document's static score set at index time (currently `DocIndex.default_score = 1`). Does not use term frequencies at all.

Requires:
- Per-document score storage (if FT.CREATE ever gains a per-doc WEIGHT parameter)
- Or just returns the index-wide `default_score` for all matches (simpler, matches current Redis behavior when no per-doc score is set)
- Plumbing `default_score` from `DocIndex` through to the scoring context

**HAMMING**
Hamming distance between query payload and document payload. Dragonfly does not support document payloads, so this scorer is out of scope until/unless payload support is added.

- Scoring documentation: https://redis.io/docs/latest/develop/ai/search-and-query/advanced-concepts/scoring/
- Current implementation: #7093 (TF storage), #7101 (BM25STD engine), #7181 (API integration)

Contributor guide

Open the contributing guide

Research direction

Start by reading the current scoring work referenced in #7093, #7101, and #7181, then inspect TakeScoredTopK, ScoreDocument, and DocIndex.default_score. Define the scoring and API decisions for BM25STD.NORM, BM25STD.TANH, DISMAX, and DOCSCORE, and verify that all four scorers return the intended results; HAMMING remains out of scope without document payloads.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.