apache / apache/lucene

Potential floating point precision error in ConjuncionScorer.score() [LUCENE-5343]

Open
#6,407 0 comments 0 reactions 0 assignees View on GitHub
affects-version:3.6.1 affects-version:3.6.3 affects-version:4.5 affects-version:4.5.1 legacy-jira-priority:Major module:core/query/scoring type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I have been investigating an issue with document scoring and found that the ConjunctionScorer implements the score method in a way that can cause floating point precision rounding issues. I noticed in some of my test cases that documents that have not been merged/optimized (I'm not sure of the correct terminology, they have a docNum of 0) have scorers added in a different order than optimized documents. Using a float to maintain the sum of scores introduces the potential for floating point precision errors. In turn this causes the score that is returned from the ConjunctionScorer to be different for some merged/unmerged documents that should have identical scores.

Example:

float sum1 = 0.0061859353f + 0.0061859353f + 0.0030929677f + 0.0030929677f + 0.0030929677f + 0.5010608f + 0.0061859353f;

float sum2 = 0.0061859353f + 0.0061859353f + 0.0061859353f + 0.0030929677f + 0.0030929677f + 0.0030929677f + 0.5010608f;

sum1 == 0.5288975; // Incorrect
sum2 == 0.52889746; // Correct

I also noticed that there is a comment in the 4.5.1 version of Lucene to the effect of:
// TODO: sum into a double and cast to float if we ever send required clauses to BS1

Is there a reason that this has not been implemented yet?

public float score() throws IOException {
double sum = 0.0d;
for (int i = 0; i < scorers.length; i++) {
sum += scorers[i].score();
}
return (float)sum;
}

---
Migrated from [LUCENE-5343](https://issues.apache.org/jira/browse/LUCENE-5343) by Jonathan Hoag

Contributor guide

Open the contributing guide

Research direction

Start by reading ConjunctionScorer.score() and the nearby comment about summing into a double. Reproduce the two score-order examples, then verify that summing scorer scores in double precision and casting at return preserves identical document scores; the issue does not name a specific test file.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.