rewrite_table_path does not update sizes of rewritten manifests in manifest lists
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.9.2 (latest release)
### Query engine
None
### Please describe the bug 🐞
Iceberg's `rewrite_table_path` procedure (`RewriteTablePath` Spark action) does not update sizes of rewritten manifests in manifest lists that reference them.
Specifically, when [`RewriteTablePathUtil.rewriteManifestList`](https://github.com/apache/iceberg/blob/95a578d7ad96aa3c82dccd305f556c83b3360528/core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java#L262) rewrites the manifest list, it copies the contained manifests and only updates their paths. [Manifest lists are rewritten first and manifest files are rewritten second](https://github.com/apache/iceberg/blob/main/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java#L289-L297), so the utility has no way of knowing final sizes of the rewritten manifest files. This is very similar to https://github.com/apache/iceberg/issues/12554.
Mismatch between manifest size in manifest list and size of the physical manifest file may cause correctness problems in some query engines. For example if the actual manifest file is larger than its size recorded in manifest lists, Trino may only read part of it and ignore the rest, effectively skipping manifest entries (referenced data and delete files) silently.
This is reproducible in Iceberg 1.8.1 at least, and based on the source code this is also true for later versions.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start with RewriteTablePathUtil.rewriteManifestList and the manifest rewrite ordering in RewriteTablePathSparkAction.java. Trace how rewritten manifest paths and physical sizes are produced, then verify that manifest-list sizes match the resulting manifest files and that the reported skipped-entry scenario is prevented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100