apache / apache/lucene

Optimize non-scoring minimum-should-match queries that fall back to WANDScorer

Open
#16,626 2 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

BooleanScorerSupplier#opt` currently selects `WANDScorer` whenever `minShouldMatch > 1`, including score modes that don't require scores:
https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/search/BooleanScorerSupplier.java#L580

For these non-scoring modes, `WANDScorer` cannot benefit from block-max score pruning. It returns candidate documents through its approximation and performs the actual minimum-should-match verification in
`TwoPhaseIterator.matches()`.

Before https://github.com/apache/lucene-solr/pull/2205, query used `MinShouldMatchSumScorer`, whose `nextDoc()` implementation searched internally for a document satisfying `minShouldMatch`. Although the two implementations are similar, this
behavior may be more efficient when scores are not required.

In our production environment, upgrading lucene make query pattern from `MinShouldMatchSumScorer` to `WANDScorer` caused more CPU consumed.

I plan to benchmark `MinShouldMatchSumScorer` and `WANDScorer` with non-scoring query to see the difference.

Contributor guide

Open the contributing guide

Research direction

Start in lucene/core/src/java/org/apache/lucene/search/BooleanScorerSupplier.java at the BooleanScorerSupplier#opt selection around line 580. Compare MinShouldMatchSumScorer and WANDScorer for minShouldMatch > 1 in non-scoring modes, focusing on their candidate and minimum-should-match verification behavior. Benchmark both implementations; done means establishing the performance difference and whether the scorer choice should change.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.