apache / apache/lucene

WANDScorer could skip more [LUCENE-10121]

Open
#11,158 1 comment 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I was looking at the NYC Taxis benchmark recently and got puzzled by the fact that the query (cab_color:y OR cab_color:g) ran so slowly: http://people.apache.org/\~mikemccand/lucenebench/sparseResults.html#search_bq_qps. This is supposed to be a best-case scenario for WAND: there are only two possible scores for documents, this query should return instantly in the sorted case.

After digging I noticed that this is due to the scaling that we due in WANDScorer to avoid floating-point rounding errors: documents can be considered as possible matches according to the scaled scores (which are rounded) while they cannot possibly match according to the actual scores. This is especially visible when many blocks contain a document that has the maximum score across the entire postings list, so any field indexed with indexOptions=DOCS or constant-scoring queries for instance.

---
Migrated from [LUCENE-10121](https://issues.apache.org/jira/browse/LUCENE-10121) by Adrien Grand (@jpountz), updated Sep 24 2021

Contributor guide

Open the contributing guide

Research direction

Reproduce the NYC Taxis benchmark query `cab_color:y OR cab_color:g` from the linked sparse-results benchmark, then inspect WANDScorer's score-scaling behavior. Done means the scorer can skip documents that cannot match according to actual scores and the sorted benchmark no longer shows the reported slowdown.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.