apache / apache/datafusion-comet

Overlap DV bitmap and Parquet footer reads during native Delta scan preparation

Open
#5,657 0 comments 0 reactions 0 assignees View on GitHub
requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Follow-up from the native Delta Lake scan review (#5365). For on-disk deletion vectors, the DV bitmap read and the Parquet footer read are independent, but preparation currently completes bitmap fetch, decode, and validation before starting the footer read. Overlapping them could reduce remote-storage preparation latency.

Any change should retain bounded concurrency across files, preserve cancellation, error handling, and memory reservations, and measure peak memory alongside latency since overlap raises concurrent resource use. Inline DVs have no bitmap network read to overlap.

### Describe the potential solution

No response

### Additional context

Raised in the #5365 review discussion.

Contributor guide

Open the contributing guide

Research direction

Start with the native Delta Lake scan preparation discussed in #5365 and trace the on-disk DV bitmap and Parquet footer read paths. Compare their current sequencing and identify how bounded per-file concurrency, cancellation, error handling, and memory reservations are enforced. Done means independent reads overlap for on-disk DVs, inline DVs remain unchanged, and latency and peak memory are measured.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.