WANDScorer could skip more [LUCENE-10121]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
I was looking at the NYC Taxis benchmark recently and got puzzled by the fact that the query (cab_color:y OR cab_color:g) ran so slowly: http://people.apache.org/\~mikemccand/lucenebench/sparseResults.html#search_bq_qps. This is supposed to be a best-case scenario for WAND: there are only two possible scores for documents, this query should return instantly in the sorted case.
After digging I noticed that this is due to the scaling that we due in WANDScorer to avoid floating-point rounding errors: documents can be considered as possible matches according to the scaled scores (which are rounded) while they cannot possibly match according to the actual scores. This is especially visible when many blocks contain a document that has the maximum score across the entire postings list, so any field indexed with indexOptions=DOCS or constant-scoring queries for instance.
---
Migrated from [LUCENE-10121](https://issues.apache.org/jira/browse/LUCENE-10121) by Adrien Grand (@jpountz), updated Sep 24 2021
Contributor guide
Research direction
Reproduce the NYC Taxis benchmark query `cab_color:y OR cab_color:g` from the linked sparse-results benchmark, then inspect WANDScorer's score-scaling behavior. Done means the scorer can skip documents that cannot match according to actual scores and the sorted benchmark no longer shows the reported slowdown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100