apache / apache/iceberg-rust

Allow `ArrowReader` to read shared `FileScanTasks`

Open
#2,964 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

The eager DataFusion introduced by #2671 scan stores planned tasks in shared `Arc<[FileScanTask]>` groups. However, `ArrowReader::read()` only accepts a stream of owned `FileScanTask`, so every task must be cloned before it can be read.

This clone copies the task's owned data and is repeated for every file in the scan.

Related discussion: https://github.com/apache/iceberg-rust/pull/2298#discussion_r3404728871

### Describe the solution you'd like

Allow `ArrowReader` to read shared `FileScanTask` without cloning their contents.

The shared input must remain alive for the lifetime of the returned stream. The existing owned-task API should remain available because `TableScan::plan_files()` can already transfer its tasks directly without cloning.

### Willingness to contribute

I would be willing to contribute to this feature with guidance from the Iceberg Rust community

Contributor guide

Open the contributing guide

Research direction

Start at ArrowReader::read and inspect how it accepts FileScanTask values and constructs the returned stream. Compare that path with TableScan::plan_files(), which can transfer owned tasks, and trace the shared-task lifetime requirement. Done means shared FileScanTasks can be read without cloning their contents while the existing owned-task API remains available.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.