apache / apache/datafusion-comet

Support splitting large DV-backed files in the native Delta scan

Open
#5,655 1 comment 0 reactions 0 assignees View on GitHub
requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Follow-up from the native Delta Lake scan review (#5365). The admitted DV path carries Spark's row-index field, which causes the shared planner to disable file splitting, so a large DV-backed file is read by one task. Enabling native DV splitting would increase scan parallelism.

If splitting is enabled, access-plan construction must prepare only the owned row groups while preserving file-global row positions, whole-file corruption validation, and memory accounting. Dictionary-page ownership, half-open split boundaries, and interaction with pruning deserve explicit regressions. Descriptor reuse can then be evaluated where multiple splits share one DV, and split-file benchmarks should accompany the change.

### Describe the potential solution

No response

### Additional context

Raised in the #5365 review discussion.

Contributor guide

Open the contributing guide

Research direction

Start with the native Delta scan and shared planner discussed in review #5365, then trace DV access-plan construction and how row-index handling disables splitting. Define regressions for row-group ownership, file-global positions, corruption validation, memory accounting, dictionary pages, split boundaries, and pruning; add split-file benchmarks before evaluating descriptor reuse.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, scala
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.