apache / apache/iceberg

Core: Stream DV Puffin rewrite in RewriteTablePathUtil to reduce memory pressure

Open
#15,924 2 comments 0 reactions 1 assignee Claimed by @aviralgarg05 View on GitHub
core good first issue improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

### Description

`RewriteTablePathUtil.rewriteDVFile` currently rewrites deletion vector (DV) Puffin files by reading all rewritten blobs into an in-memory list before writing them out, which can create unnecessary memory pressure when rewriting larger DV files or files with many blobs.

### Suggested improvement

Refactor `RewriteTablePathUtil.rewriteDVFile` to stream rewritten blobs directly to the output `PuffinWriter` while iterating through the source blobs, instead of collecting all rewritten blobs into a temporary list first.

### Query engine

None

### Willingness to contribute

- [ ] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [x] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.