LUCENE-9071: Speed up BM25 scores. (#1043)
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
https://github.com/apache/lucene/blob/0345fcabb3ceb71fb39c1aa77502bae220f259ca/lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java#L246,How to understand monotonicity?, freq * 1/norm is guaranteed to be monotonic? but My experimental results are the same as freq / (freq + norm).
the code is
@Test
public void test() {
int length = 2;
float freq = 1.0f;
float[] cache = {16777216.0f, 16777217.0f};
float[] inverseCache = new float[length];
for (int i = 0; i < length; i++) {
inverseCache[i] = 1f / cache[i];
}
for (int i = 0; i < length; i++) {
System.out.println(freq / (freq + cache[i]));
}
for (int i = 0; i < length; i++) {
System.out.println(freq * inverseCache[i]);
}
}
Contributor guide
Research direction
Start in lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java around line 246 and compare the two expressions in the issue's test snippet. Run the provided floating-point experiment and inspect nearby BM25 similarity tests. Done means the monotonicity question is resolved and the implementation or regression coverage reflects the confirmed behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100