Skip redundant SplitScanTask for single-split files
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
BaseContentScanTask.split() wraps every splittable file in a SplitScanTask even when the file fits within a single split (fixed-size path, no row-group offsets). For tables with many small files this allocates one redundant wrapper object per data file during scan planning, adding to driver heap pressure.
### Query engine
Spark
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Start at BaseContentScanTask.split() and inspect how it creates SplitScanTask for fixed-size files without row-group offsets. Confirm the single-split case can avoid the wrapper while multi-split files retain it. Done means scan planning no longer allocates the redundant wrapper for single-split files, with existing behavior preserved otherwise.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100