apache / apache/iceberg

Snapshot Changes should Support Multiple Snapshots

Open
#15,660 2 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

Currently SnapshotChanges only lets you find the changes from 1 snapshot, but we should probably allow it to find changes from many snapshots since I think we can probably parallelize a bit better than a user with individual snapshot calls. We could also centralize the streaming code as I noted in #15659 giving a single point for interacting with deltas between snapshots.

### Query engine

None

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start with the SnapshotChanges entry point and read the related discussion in #15659. Define how multiple snapshots should be accepted and how streaming between snapshots should be centralized, then verify that the resulting API can find changes across multiple snapshots and preserves the intended parallelization behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.