apache / apache/datafusion-comet
Support splitting large DV-backed files in the native Delta scan
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Follow-up from the native Delta Lake scan review (#5365). The admitted DV path carries Spark's row-index field, which causes the shared planner to disable file splitting, so a large DV-backed file is read by one task. Enabling native DV splitting would increase scan parallelism.
If splitting is enabled, access-plan construction must prepare only the owned row groups while preserving file-global row positions, whole-file corruption validation, and memory accounting. Dictionary-page ownership, half-open split boundaries, and interaction with pruning deserve explicit regressions. Descriptor reuse can then be evaluated where multiple splits share one DV, and split-file benchmarks should accompany the change.
### Describe the potential solution
No response
### Additional context
Raised in the #5365 review discussion.
Contributor guide
Research direction
Start with the native Delta scan and shared planner discussed in review #5365, then trace DV access-plan construction and how row-index handling disables splitting. Define regressions for row-group ownership, file-global positions, corruption validation, memory accounting, dictionary pages, split boundaries, and pruning; add split-file benchmarks before evaluating descriptor reuse.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, scala
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100