Enable parallel file-level scanning for IcebergTableScan Datafusion Integration
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Is your feature request related to a problem or challenge?
As noted in [#1604](https://github.com/apache/iceberg-rust/issues/1604), Iceberg-DataFusion read performance is currently bottlenecked by single-threaded execution. While [size-based planning](https://github.com/apache/iceberg-rust/issues/128) is the proposed long-term solution, a more immediate improvement would be to parallelize over FileScanTask and leverage `ArrowReaderBuilder` during plan execution.
### Describe the solution you'd like
Pre-calculates the `FileScanTask` streams and partitions them across the available DataFusion partitions, updating the IcebergTableScan struct and ExecutionPlan trait:
**Pre-partitioning Scan Tasks:** IcebergTableScan now accepts a grouped tasks: Vec> rather than computing streams eagerly.
**Propagating Partition Counts:** The compute_properties method now dynamically returns Partitioning::UnknownPartitioning(tasks.len()) instead of the hardcoded 1.
**Parallel Stream Execution:** The execute phase uses `self.tasks.get(partition)` to spawn an `ArrowReaderBuilder` specific only to the slice of tasks mapped to that discrete DataFusion partition index.
### Willingness to contribute
I would be willing to contribute to this feature with guidance from the Iceberg Rust community
Contributor guide
Research direction
Read the IcebergTableScan compute_properties and execute paths, along with FileScanTask and ArrowReaderBuilder usage. Verify that scan tasks are grouped by DataFusion partition, partition counts are reported dynamically, and each execution partition reads only its assigned tasks in parallel.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100