apache / apache/lucene

LUCENE-9071: Speed up BM25 scores. (#1043)

Open
#13,299 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

https://github.com/apache/lucene/blob/0345fcabb3ceb71fb39c1aa77502bae220f259ca/lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java#L246,How to understand monotonicity?, freq * 1/norm is guaranteed to be monotonic? but My experimental results are the same as freq / (freq + norm).
the code is
@Test
public void test() {
int length = 2;
float freq = 1.0f;
float[] cache = {16777216.0f, 16777217.0f};
float[] inverseCache = new float[length];
for (int i = 0; i < length; i++) {
inverseCache[i] = 1f / cache[i];
}
for (int i = 0; i < length; i++) {
System.out.println(freq / (freq + cache[i]));
}
for (int i = 0; i < length; i++) {
System.out.println(freq * inverseCache[i]);
}
}

Contributor guide

Open the contributing guide

Research direction

Start in lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java around line 246 and compare the two expressions in the issue's test snippet. Run the provided floating-point experiment and inspect nearby BM25 similarity tests. Done means the monotonicity question is resolved and the implementation or regression coverage reflects the confirmed behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.