apache / apache/lucene

Efficient iteration over deleted doc values

Open
#15,226 5 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

As part of https://github.com/apache/lucene/pull/14439, we introduced efficient histogram collection using PointTrees. Unfortunately, the optimization falls apart even with single deleted document in a segment. I am wondering if there is a way to efficiently iterate over deleted doc values and correct the values for each bucket.

I had thought about it a while back (and kind of forgot) and @mikemccand brought it up again during my talk on *Efficient Histogram Collection in Lucene with BulkCollector* at recently concluded Apache Community over Code

Contributor guide

Open the contributing guide

Research direction

Start by reading PR 14439 and the PointTrees and BulkCollector changes described in this issue, then trace how histogram collection handles deleted documents. Investigate whether deleted doc values can be iterated efficiently and define the bucket-correction path. Done means the optimization remains effective with deleted documents while bucket values are corrected accurately.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.