apache / apache/iceberg-rust

Add snapshot_id parameter for Datafusion IcebergTableProvider (Distributed reads)

Open
#2,861 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

Currently table scans are pinned to current snapshot for catalog backed providers
`
Ok(Arc::new(IcebergTableScan::new(
table,
None, // Always use current snapshot for catalog-backed provider
self.schema.clone(),
projection,
filters,
limit,
)))
`
This is fine for single-node reads, but for a distributed engine, each remote node must read against the same snapshot_id the coordinator planned against. Otherwise a concurrent write during reads can result in worker nodes reading incorrect versions of the catalog.

related to https://github.com/apache/iceberg-rust/pull/2613

### Describe the solution you'd like

add a passable snapshot_id to IcebergTableScan to pin reads

### Willingness to contribute

yes

Contributor guide

Open the contributing guide

Research direction

Start with the IcebergTableProvider scan construction shown in the issue and trace IcebergTableScan, then review related PR #2613. Determine how a planned snapshot_id reaches distributed reads; done means callers can pass it and remote workers read the same snapshot instead of the current catalog state.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.