Nested spanNear scoring error when inner clauses overlap positions [LUCENE-7151]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
For spanNear([spanNear([contents:word1, contents:word3], 2, true), spanNear([contents:word2, contents:word3], 2, true)], 2, false)
Scores for the following two documents should be the same but are not.
doc1: [----- word1 word3 ----- word2 word3 ----- word1 word2 word3 -----]
doc2: [----- word2 word3 ----- word1 word3 ----- word1 word2 word3 -----]
The positions of the inner clauses effect the scoring for the of the final 3-term phrase. This appears to be a side-effect of the span-scoring rewrite in 5.2(?).
NearSpansUnordered's SpansCell.adjustMax() uses end-position values to decide maxEndPositionCell while the SpanPositionQueue uses start-position and end-position values to sort the SpanCells. This means that maxEndPositionCell will be incorrectly set or not set depending on previous positions.
I can provide example code illustrating the score error.
---
Migrated from [LUCENE-7151](https://issues.apache.org/jira/browse/LUCENE-7151) by David Wendt
Environment:
```
Windows, Linux
```
Attachments: [SpanScore5Bug.java](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-7151/SpanScore5Bug.java)
Contributor guide
Research direction
Start by reviewing the attached SpanScore5Bug.java reproduction and the span scoring code around NearSpansUnordered's SpansCell.adjustMax() and SpanPositionQueue. Compare the scores for the two document layouts and trace how start and end positions select maxEndPositionCell. Done means both documents receive the same score for the described query.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100