apache / apache/datafusion-comet
Preserve compressed bitmap containers when decoding native-format deletion vectors
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Follow-up from the native Delta Lake scan review (#5365). The native-format deletion vector decoder enumerates each deleted position and reinserts it into a treemap, paying an extra pass. Preserving the compressed bitmap containers during conversion would avoid it. Portable-format decoding already deserializes directly.
Evaluation should cover sparse, contiguous, and alternating deletion patterns, measuring decoding time and allocations. This does not remove the later iteration needed for partial row group selections.
### Describe the potential solution
A candidate patch preserving sub-bitmap containers via from_bitmaps exists and can be adapted once #5365 lands.
### Additional context
Raised in the #5365 review discussion.
Contributor guide
Research direction
Start by reading the native-format deletion vector decoder and the candidate from_bitmaps patch, then compare their behavior with portable-format decoding after #5365 lands. Evaluate sparse, contiguous, and alternating deletion patterns for decoding time and allocations; done means compressed bitmap containers are preserved without removing the later partial-row-group iteration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, scala
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100