Use intoBitSet to speed up ReqExclBulkScorer with dense exclusions
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
`ReqExclBulkScorer` currently processes prohibited documents using a leapfrog loop. It scores required documents up to the next excluded document, advances the prohibited iterator, and repeats. This works well for sparse prohibited clauses. However, it becomes expensive when the prohibited side is dense, especially when it is a disjunction of multiple high-frequency terms.
`docIDRunEnd()` only helps when the iterator can expose a consecutive run of matching document IDs. Dense disjunctions are often composed of interleaved postings rathe. In these cases, `docIDRunEnd()` frequently returns only `docID()` + 1.
I'm planing to introduce a dense path that collects prohibited documents into 4096-document `bitset` windows, applies `acceptDocs`, and scores the required clause once per window.
Contributor guide
Research direction
Start by reading ReqExclBulkScorer and locating the existing docIDRunEnd() leapfrog path, then inspect how intoBitSet and acceptDocs are used. Done means dense prohibited disjunctions can be processed in 4096-document windows while preserving required scoring and document acceptance behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance, search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100