Optimize filtering on the primary index sort field
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Filtering on the primary index sort field is already quite efficient, but we could do better. E.g. consider the following query `+description:(Apache Lucene) #category:books`, assuming an index sorted by `category`.
The most efficient way to evaluate this query would consist of computing the range of doc IDs that the `category: books` filter matches, and then evaluate the `description:(Apache Lucene)` query using it bulk scorer (`MaxScoreBulkScorer` in this case) on this range of doc IDs instead of the whole doc ID space.
My suggestion for implementing this would consist of:
- Introducing a new `FilteredOnPrimaryIndexSortFieldQuery`, not exposed to users
- Rewriting `BooleanQuery` to a `FilteredOnPrimaryIndexSortFieldQuery` that wraps the same `BooleanQuery` minus the filter on the primary sort field when applicable
- Making `FilteredOnPrimaryIndexSortFieldQuery`'s bulk scorer compute the range of doc IDs matched by the filter at creation time, and then intersect ranges of doc IDs passed to `BulkScorer#score` with the range of doc IDs matched by the filter before delegating.
This would be quite nice as many queries would perform as fast as if they were running in a Lucene index that only contained documents matched by the filter.
Contributor guide
Research direction
Start by reading BooleanQuery, BulkScorer, and MaxScoreBulkScorer, then trace how primary index sort fields expose matching document-ID ranges. The proposed work introduces an internal FilteredOnPrimaryIndexSortFieldQuery and applies it when a BooleanQuery includes a filter on the primary sort field. Done means the bulk scorer evaluates the remaining query only within the filter's matching ranges.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100