apache / apache/iceberg

Skip redundant SplitScanTask for single-split files

Open
#17,999 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

BaseContentScanTask.split() wraps every splittable file in a SplitScanTask even when the file fits within a single split (fixed-size path, no row-group offsets). For tables with many small files this allocates one redundant wrapper object per data file during scan planning, adding to driver heap pressure.

### Query engine

Spark

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start at BaseContentScanTask.split() and inspect how it creates SplitScanTask for fixed-size files without row-group offsets. Confirm the single-split case can avoid the wrapper while multi-split files retain it. Done means scan planning no longer allocates the redundant wrapper for single-split files, with existing behavior preserved otherwise.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.