Add snapshot_id parameter for Datafusion IcebergTableProvider (Distributed reads)
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Is your feature request related to a problem or challenge?
Currently table scans are pinned to current snapshot for catalog backed providers
`
Ok(Arc::new(IcebergTableScan::new(
table,
None, // Always use current snapshot for catalog-backed provider
self.schema.clone(),
projection,
filters,
limit,
)))
`
This is fine for single-node reads, but for a distributed engine, each remote node must read against the same snapshot_id the coordinator planned against. Otherwise a concurrent write during reads can result in worker nodes reading incorrect versions of the catalog.
related to https://github.com/apache/iceberg-rust/pull/2613
### Describe the solution you'd like
add a passable snapshot_id to IcebergTableScan to pin reads
### Willingness to contribute
yes
Contributor guide
Research direction
Start with the IcebergTableProvider scan construction shown in the issue and trace IcebergTableScan, then review related PR #2613. Determine how a planned snapshot_id reaches distributed reads; done means callers can pass it and remote workers read the same snapshot instead of the current catalog state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100