apache / apache/iceberg

Snapshot Changes should have a Streaming Interface

Open
#15,659 1 comment 0 reactions 0 assignees View on GitHub
improvement stale
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

Currently `SnapshotChanges` only provides a caching interface that eagerly materializes all file changes into in-memory lists and returns `Iterable`. There are use cases where returning a streaming `CloseableIterable` would be preferable see #13556 for example.

I think we probably should just add CloseableIterable interfaces to SnapshotChanges as well and just have the caching calls wrap them.

### Query engine

None

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start by reading the SnapshotChanges class and the CloseableIterable usage described by example #13556. Trace how SnapshotChanges currently materializes changes into Iterable results, then verify that streaming interfaces are available and the existing caching calls wrap them without changing their behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.