apache / apache/lucene

Use intoBitSet to speed up ReqExclBulkScorer with dense exclusions

Open
#16,668 0 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

`ReqExclBulkScorer` currently processes prohibited documents using a leapfrog loop. It scores required documents up to the next excluded document, advances the prohibited iterator, and repeats. This works well for sparse prohibited clauses. However, it becomes expensive when the prohibited side is dense, especially when it is a disjunction of multiple high-frequency terms.

`docIDRunEnd()` only helps when the iterator can expose a consecutive run of matching document IDs. Dense disjunctions are often composed of interleaved postings rathe. In these cases, `docIDRunEnd()` frequently returns only `docID()` + 1.

I'm planing to introduce a dense path that collects prohibited documents into 4096-document `bitset` windows, applies `acceptDocs`, and scores the required clause once per window.

Contributor guide

Open the contributing guide

Research direction

Start by reading ReqExclBulkScorer and locating the existing docIDRunEnd() leapfrog path, then inspect how intoBitSet and acceptDocs are used. Done means dense prohibited disjunctions can be processed in 4096-document windows while preserving required scoring and document acceptance behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance, search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.