apache / apache/lucene

Enforcing BooleanClause.Occur.FILTER before CustomScoreQuery [LUCENE-7443]

Open
#8,495 0 comments 0 reactions 0 assignees View on GitHub
affects-version:6.2 legacy-jira-priority:Minor type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I have a BooleanQuery including a BooleanClause.Occur.MUST clause and a BooleanClause.Occur.FILTER clause. I then use the BooleanQuery in a CustomScoreQuery. It seems that the filtering constraint is not enforced before calculating the score in CustomScoreQuery and is enforced after the calculation. This can cause performance issues, since CustomScoreQuery may perform costly operations to rescore the BooleanQuery results. If the filter is enforced before rescoring, much less documents may require rescoring in CustomScoreQuery, and it does not make sense to spend a lot of processing for documents we know will be filtered out.
More details:
The BooleanQuery is used as subQuery in the following constructor, and a costly scoringQuery is used which is more rational to calculate it only when the results of subQuery is not trivially filtered out because of the FILTER in subQuery.
public CustomScoreQuery(Query subQuery, FunctionQuery scoringQuery) {

---
Migrated from [LUCENE-7443](https://issues.apache.org/jira/browse/LUCENE-7443) by Shayan Tabrizi

Contributor guide

Open the contributing guide

Research direction

Start at the CustomScoreQuery(Query subQuery, FunctionQuery scoringQuery) constructor and trace how its BooleanQuery subQuery handles MUST and FILTER clauses. Determine where filtering occurs relative to the costly scoringQuery calculation. Done means FILTER constraints are applied before rescoring, avoiding work for documents that will be filtered out.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.