Core: Stream DV Puffin rewrite in RewriteTablePathUtil to reduce memory pressure
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
### Description
`RewriteTablePathUtil.rewriteDVFile` currently rewrites deletion vector (DV) Puffin files by reading all rewritten blobs into an in-memory list before writing them out, which can create unnecessary memory pressure when rewriting larger DV files or files with many blobs.
### Suggested improvement
Refactor `RewriteTablePathUtil.rewriteDVFile` to stream rewritten blobs directly to the output `PuffinWriter` while iterating through the source blobs, instead of collecting all rewritten blobs into a temporary list first.
### Query engine
None
### Willingness to contribute
- [ ] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [x] I cannot contribute this improvement/feature at this time
Contributor guide
Assessment
This issue has not been assessed yet.