Allow `ArrowReader` to read shared `FileScanTasks`
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Is your feature request related to a problem or challenge?
The eager DataFusion introduced by #2671 scan stores planned tasks in shared `Arc<[FileScanTask]>` groups. However, `ArrowReader::read()` only accepts a stream of owned `FileScanTask`, so every task must be cloned before it can be read.
This clone copies the task's owned data and is repeated for every file in the scan.
Related discussion: https://github.com/apache/iceberg-rust/pull/2298#discussion_r3404728871
### Describe the solution you'd like
Allow `ArrowReader` to read shared `FileScanTask` without cloning their contents.
The shared input must remain alive for the lifetime of the returned stream. The existing owned-task API should remain available because `TableScan::plan_files()` can already transfer its tasks directly without cloning.
### Willingness to contribute
I would be willing to contribute to this feature with guidance from the Iceberg Rust community
Contributor guide
Research direction
Start at ArrowReader::read and inspect how it accepts FileScanTask values and constructs the returned stream. Compare that path with TableScan::plan_files(), which can transfer owned tasks, and trace the shared-task lifetime requirement. Done means shared FileScanTasks can be read without cloning their contents while the existing owned-task API remains available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100