Snapshot Changes should have a Streaming Interface
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
Currently `SnapshotChanges` only provides a caching interface that eagerly materializes all file changes into in-memory lists and returns `Iterable`. There are use cases where returning a streaming `CloseableIterable` would be preferable see #13556 for example.
I think we probably should just add CloseableIterable interfaces to SnapshotChanges as well and just have the caching calls wrap them.
### Query engine
None
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Start by reading the SnapshotChanges class and the CloseableIterable usage described by example #13556. Trace how SnapshotChanges currently materializes changes into Iterable results, then verify that streaming interfaces are available and the existing caching calls wrap them without changing their behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100