apache / apache/lucene

Optimize filtering on the primary index sort field

Open
#15,139 8 comments 0 reactions 0 assignees View on GitHub
type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Filtering on the primary index sort field is already quite efficient, but we could do better. E.g. consider the following query `+description:(Apache Lucene) #category:books`, assuming an index sorted by `category`.

The most efficient way to evaluate this query would consist of computing the range of doc IDs that the `category: books` filter matches, and then evaluate the `description:(Apache Lucene)` query using it bulk scorer (`MaxScoreBulkScorer` in this case) on this range of doc IDs instead of the whole doc ID space.

My suggestion for implementing this would consist of:
- Introducing a new `FilteredOnPrimaryIndexSortFieldQuery`, not exposed to users
- Rewriting `BooleanQuery` to a `FilteredOnPrimaryIndexSortFieldQuery` that wraps the same `BooleanQuery` minus the filter on the primary sort field when applicable
- Making `FilteredOnPrimaryIndexSortFieldQuery`'s bulk scorer compute the range of doc IDs matched by the filter at creation time, and then intersect ranges of doc IDs passed to `BulkScorer#score` with the range of doc IDs matched by the filter before delegating.

This would be quite nice as many queries would perform as fast as if they were running in a Lucene index that only contained documents matched by the filter.

Contributor guide

Open the contributing guide

Research direction

Start by reading BooleanQuery, BulkScorer, and MaxScoreBulkScorer, then trace how primary index sort fields expose matching document-ID ranges. The proposed work introduces an internal FilteredOnPrimaryIndexSortFieldQuery and applies it when a BooleanQuery includes a filter on the primary sort field. Done means the bulk scorer evaluates the remaining query only within the filter's matching ranges.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.