Reduce the number of classes extending DocIdSetIterator
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Megamorphism of call sites to `DocIdSetIterator#nextDoc` and `DocIdSetIterator#advance` hurts search performance. See e.g. annotation HJ at https://benchmarks.mikemccandless.com/OrHighHigh.html where reducing polymorphism helped improve QPS by 40%. More specialization could also help, but before we start going this route, we should reduce the number of classes that extend `DocIdSetIterator` so that specialization is less needed and can focus on some key impls.
- [x] https://github.com/apache/lucene/issues/7977
- [x] https://github.com/apache/lucene/pull/14017
- [x] https://github.com/apache/lucene/pull/14033
- [x] https://github.com/apache/lucene/pull/14451
- [ ] https://github.com/apache/lucene/pull/14475
- [ ] Make `IntervalIterator` expose a `DocIdSetIterator` instead of extending `DocIdSetIterator`.
- [ ] Make `Spans` expose a `DocIdSetIterator` instead of extending `DocIdSetIterator`.
- [ ] Can we somehow remove `ImpactsDISI`, e.g. by evaluating term queries as single-clause disjunctions or conjunctions?
Contributor guide
Research direction
Start by reviewing the checked items and the remaining proposals around IntervalIterator, Spans, and ImpactsDISI, then trace their DocIdSetIterator relationships. Done means reducing the number of extending classes through the listed API changes or another agreed approach, while preserving search behavior and addressing the performance goal.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100