apache / apache/iceberg-rust

Enable parallel file-level scanning for IcebergTableScan Datafusion Integration

Open
#2,220 3 comments 9 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

As noted in [#1604](https://github.com/apache/iceberg-rust/issues/1604), Iceberg-DataFusion read performance is currently bottlenecked by single-threaded execution. While [size-based planning](https://github.com/apache/iceberg-rust/issues/128) is the proposed long-term solution, a more immediate improvement would be to parallelize over FileScanTask and leverage `ArrowReaderBuilder` during plan execution.

### Describe the solution you'd like

Pre-calculates the `FileScanTask` streams and partitions them across the available DataFusion partitions, updating the IcebergTableScan struct and ExecutionPlan trait:

**Pre-partitioning Scan Tasks:** IcebergTableScan now accepts a grouped tasks: Vec> rather than computing streams eagerly.
**Propagating Partition Counts:** The compute_properties method now dynamically returns Partitioning::UnknownPartitioning(tasks.len()) instead of the hardcoded 1.
**Parallel Stream Execution:** The execute phase uses `self.tasks.get(partition)` to spawn an `ArrowReaderBuilder` specific only to the slice of tasks mapped to that discrete DataFusion partition index.

### Willingness to contribute

I would be willing to contribute to this feature with guidance from the Iceberg Rust community

Contributor guide

Open the contributing guide

Research direction

Read the IcebergTableScan compute_properties and execute paths, along with FileScanTask and ArrowReaderBuilder usage. Verify that scan tasks are grouped by DataFusion partition, partition counts are reported dynamically, and each execution partition reads only its assigned tasks in parallel.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.