apache / apache/iceberg-rust

[iceberg-datafusion] Cache eager scan planning across equivalent scans

Open
#2,963 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

Eager scan planning introduced by #2671 calls `plan_files()` during `TableProvider::scan()`. This includes reading manifests, binding predicates, resolving the snapshot schema, and associating applicable delete files with each FileScanTask.

If DataFusion requests the same scan multiple times during optimization, this work is repeated. The resulting eager plan is retained only by the returned `IcebergTableScan` and cannot be reused by a future equivalent `scan()` call.

This limitation was originally discussed in #2298: https://github.com/apache/iceberg-rust/pull/2298#discussion_r3404722001

We should keep in mind that there are two provider types:
- `IcebergStaticTableProvider` uses a fixed table and snapshot, making equivalent scans suitable for reuse.
- `IcebergTableProvider` must continue loading fresh table metadata so that newly committed snapshots remain visible.

### Describe the solution you'd like

Reuse eager scan planning results across equivalent `TableProvider::scan()` calls.

A cached result must only be reused when every input affecting file planning is equivalent. This includes at least:
- the concrete snapshot ID;
- the projection;
- the pushed-down predicate.

For the catalog-backed `IcebergTableProvider`, the table must still be reloaded before a cached result can be reused. The concrete snapshot ID resolved from the refreshed table must participate in cache lookup so that a new snapshot naturally causes a cache miss.

Two caching strategies should be considered:

#### Option 1: Cache the complete `EagerScanPlan`

Cache the tasks after they have been grouped into DataFusion output partitions.

#### Option 2: Cache the planned tasks before grouping

Cache the result of Iceberg file planning before assigning the `FileScanTask`s to DataFusion output partitions. Perform the relatively inexpensive grouping after cache lookup.

In this design, `target_partitions` does not need to be part of the cache key.

### Willingness to contribute

I would be willing to contribute to this feature with guidance from the Iceberg Rust community

Contributor guide

Open the contributing guide

Research direction

Start at the TableProvider::scan() implementations for IcebergStaticTableProvider and IcebergTableProvider, then trace plan_files(), EagerScanPlan, and FileScanTask. Compare the two providers' snapshot-loading behavior and choose a cache boundary and key covering snapshot ID, projection, and predicate. Done means equivalent scans reuse planning results while refreshed catalog metadata and new snapshots cause cache misses.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.