apache / apache/lucene

BooleanScorer should sometimes be used for MUST clauses [LUCENE-4396]

Open
#5,462 92 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Today we only use BooleanScorer if the query consists of SHOULD and MUST_NOT.

If there is one or more MUST clauses we always use BooleanScorer2.

But I suspect that unless the MUST clauses have very low hit count compared to the other clauses, that BooleanScorer would perform better than BooleanScorer2. BooleanScorer still has some vestiges from when it used to handle MUST so it shouldn't be hard to bring back this capability ... I think the challenging part might be the heuristics on when to use which (likely we would have to use firstDocID as proxy for total hit count).

Likely we should also have BooleanScorer sometimes use .advance() on the subs in this case, eg if suddenly the MUST clause skips 1000000 docs then you want to .advance() all the SHOULD clauses.

I won't have near term time to work on this so feel free to take it if you are inspired!

---
Migrated from [LUCENE-4396](https://issues.apache.org/jira/browse/LUCENE-4396) by Michael McCandless (@mikemccand), updated Aug 18 2014
Attachments: [all.perf](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4396/all.perf), [And.tasks](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4396/And.tasks) (versions: 3), [AndOr.tasks](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4396/AndOr.tasks) (versions: 2), [LUCENE-4396.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4396/LUCENE-4396.patch) (versions: 16), [LUCENE-4396-simple.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4396/LUCENE-4396-simple.patch) (versions: 4), [luceneutil-score-equal.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4396/luceneutil-score-equal.patch) (versions: 2), [merge.perf](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4396/merge.perf), [merge.png](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4396/merge.png), [merge-simple.perf](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4396/merge-simple.perf), [merge-simple.png](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4396/merge-simple.png), [perf.png](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4396/perf.png), [SIZE.perf](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4396/SIZE.perf), [stat.cpp](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4396/stat.cpp) (versions: 2), [tasks.cpp](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4396/tasks.cpp) (versions: 2)

Contributor guide

Open the contributing guide

Research direction

Start by reading the BooleanScorer and BooleanScorer2 implementations and the issue's attached performance data. Determine when MUST clauses should select BooleanScorer and when MUST clauses should trigger advance() on SHOULD clauses; done means the heuristic is implemented and performance and scoring behavior are validated.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.