apache / apache/iceberg-rust

Support incremental reads between snapshot-ids

Open
#2,152 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

## Feature Request: Incremental Snapshot Scanning (from/to snapshot-id)

### Is your feature request related to a problem? Please describe.

Currently, iceberg-rust only supports reading a table at a specific snapshot (time-travel) or the current snapshot. There is no way to read only the data that was added between two snapshots.

The Java Iceberg client supports this via `IncrementalDataTableScan`:

```java
// Java API
TableScan scan = table.newIncrementalDataTableScan()
.fromSnapshotExclusive(100)
.toSnapshot(200);
```

This is a critical feature for:
- **Change Data Capture (CDC)** - Reading only new/changed data for downstream systems
- **Incremental ETL pipelines** - Processing only new data since the last checkpoint
- **Efficient data synchronization** - Syncing only deltas between systems
- **Streaming workloads** - Reading appends as they happen

### Describe the solution you'd like

Add incremental scan support to `TableScanBuilder` with methods similar to the Java client:

```rust
// Scan changes between two snapshots (from exclusive, to inclusive)
let scan = table.scan()
.from_snapshot_exclusive(from_id)
.to_snapshot(to_id)
.build()?;

// Scan with inclusive from
let scan = table.scan()
.from_snapshot_inclusive(from_id)
.to_snapshot(to_id)
.build()?;

// Convenience methods
let scan = table.scan().appends_after(from_id).build()?;
let scan = table.scan().appends_between(from_id, to_id).build()?;
```

Additionally, expose this feature through the DataFusion integration:

```rust
// DataFusion integration
let provider = IcebergStaticTableProvider::try_new_incremental(table, from_id, to_id).await?;
ctx.register_table("changes", Arc::new(provider))?;
let df = ctx.sql("SELECT * FROM changes").await?;
```

### Implementation Notes

Based on the Java implementation (`IncrementalDataTableScan.java`):

1. **Snapshot Range Validation** - Walk the snapshot ancestry chain to validate that `from_snapshot` is an ancestor of `to_snapshot`

2. **Manifest Entry Filtering** - Only include manifest entries where:
- `status == ADDED` (not EXISTING or DELETED)
- `snapshot_id` is within the specified range

3. **Operation Validation** - Initially only support `APPEND` operations (same as Java). `OVERWRITE` and `DELETE` operations require additional handling for delete files.

4. **Mutual Exclusivity** - The `snapshot_id()` method (for time-travel) should be mutually exclusive with incremental scan methods.

### Describe alternatives you've considered

1. **Post-filtering** - Users could scan the full table and filter by `_snapshot_id` metadata column, but this is inefficient as it scans all data.

2. **Manual manifest parsing** - Users could manually read manifests and filter entries, but this defeats the purpose of having a scan API.

### Additional context

- Java implementation reference: `IncrementalDataTableScan.java` in apache/iceberg
- This feature is commonly requested for building CDC pipelines with Iceberg
- Spark's `incrementalAppend` reader option provides similar functionality

### Willingness to contribute

Yes

Contributor guide

Open the contributing guide

Research direction

Start with TableScanBuilder and the DataFusion IcebergStaticTableProvider integration, then compare their current snapshot and time-travel behavior with Java's IncrementalDataTableScan.java. Implement validated ancestor ranges, APPEND-only ADDED manifest filtering, and mutual exclusivity with snapshot_id(); done means the Rust and DataFusion APIs support the requested incremental scans.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.