apache / apache/datafusion-comet
Overlap DV bitmap and Parquet footer reads during native Delta scan preparation
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Follow-up from the native Delta Lake scan review (#5365). For on-disk deletion vectors, the DV bitmap read and the Parquet footer read are independent, but preparation currently completes bitmap fetch, decode, and validation before starting the footer read. Overlapping them could reduce remote-storage preparation latency.
Any change should retain bounded concurrency across files, preserve cancellation, error handling, and memory reservations, and measure peak memory alongside latency since overlap raises concurrent resource use. Inline DVs have no bitmap network read to overlap.
### Describe the potential solution
No response
### Additional context
Raised in the #5365 review discussion.
Contributor guide
Research direction
Start with the native Delta Lake scan preparation discussed in #5365 and trace the on-disk DV bitmap and Parquet footer read paths. Compare their current sequencing and identify how bounded per-file concurrency, cancellation, error handling, and memory reservations are enforced. Done means independent reads overlap for on-disk DVs, inline DVs remain unchanged, and latency and peak memory are measured.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100