Efficient iteration over deleted doc values
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
As part of https://github.com/apache/lucene/pull/14439, we introduced efficient histogram collection using PointTrees. Unfortunately, the optimization falls apart even with single deleted document in a segment. I am wondering if there is a way to efficiently iterate over deleted doc values and correct the values for each bucket.
I had thought about it a while back (and kind of forgot) and @mikemccand brought it up again during my talk on *Efficient Histogram Collection in Lucene with BulkCollector* at recently concluded Apache Community over Code
Contributor guide
Research direction
Start by reading PR 14439 and the PointTrees and BulkCollector changes described in this issue, then trace how histogram collection handles deleted documents. Investigate whether deleted doc values can be iterated efficiently and define the bucket-correction path. Done means the optimization remains effective with deleted documents while bucket values are corrected accurately.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100