[iceberg-datafusion] Balance eager scan task groups by estimated read size
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Is your feature request related to a problem or challenge?
#2671 introduces eager scan planning for the DataFusion integration. The planned `FileScanTask` are currently assigned to DataFusion output partitions using round-robin.
This balances the number of tasks per partition, but not the amount of data to read. When file sizes vary significantly, one partition may receive most of the bytes and become the query bottleneck.
This issue concerns distributing already planned tasks between DataFusion partitions. Generic file splitting and scan-task planning remain tracked by #128.
### Describe the solution you'd like
Replace the round-robin grouping in `scan_planning.rs` with size-aware grouping.
The implementation should:
- create at most `target_partitions` non-empty groups;
- balance the estimated bytes to read across those groups;
- preserve the current behavior for empty task lists and when there are fewer tasks than target partitions;
The task weight should represent the bytes scanned by the task, including the case where a task covers only part of a file.
Related to #128, but does not replace or close it: #128 covers generic size-based scan-task planning and file splitting in the core Iceberg crate.
### Willingness to contribute
I can contribute to this feature independently
Contributor guide
Research direction
Start in scan_planning.rs at the existing round-robin grouping for planned FileScanTask values, and trace how target_partitions and task ranges are represented. Run the existing iceberg-datafusion scan-planning tests, if present, before changing behavior. Done means at most target_partitions non-empty groups have balanced estimated read bytes, including partial-file tasks, while empty and undersized task lists retain their current behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100