apache / apache/lucene

Reduce the number of classes extending DocIdSetIterator

Open
#14,450 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Megamorphism of call sites to `DocIdSetIterator#nextDoc` and `DocIdSetIterator#advance` hurts search performance. See e.g. annotation HJ at https://benchmarks.mikemccandless.com/OrHighHigh.html where reducing polymorphism helped improve QPS by 40%. More specialization could also help, but before we start going this route, we should reduce the number of classes that extend `DocIdSetIterator` so that specialization is less needed and can focus on some key impls.

- [x] https://github.com/apache/lucene/issues/7977
- [x] https://github.com/apache/lucene/pull/14017
- [x] https://github.com/apache/lucene/pull/14033
- [x] https://github.com/apache/lucene/pull/14451
- [ ] https://github.com/apache/lucene/pull/14475
- [ ] Make `IntervalIterator` expose a `DocIdSetIterator` instead of extending `DocIdSetIterator`.
- [ ] Make `Spans` expose a `DocIdSetIterator` instead of extending `DocIdSetIterator`.
- [ ] Can we somehow remove `ImpactsDISI`, e.g. by evaluating term queries as single-clause disjunctions or conjunctions?

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the checked items and the remaining proposals around IntervalIterator, Spans, and ImpactsDISI, then trace their DocIdSetIterator relationships. Done means reducing the number of extending classes through the listed API changes or another agreed approach, while preserving search behavior and addressing the performance goal.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.