apache / apache/iceberg

rewrite_table_path does not update sizes of rewritten manifests in manifest lists

Open
#13,719 5 comments 5 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.9.2 (latest release)

### Query engine

None

### Please describe the bug 🐞

Iceberg's `rewrite_table_path` procedure (`RewriteTablePath` Spark action) does not update sizes of rewritten manifests in manifest lists that reference them.

Specifically, when [`RewriteTablePathUtil.rewriteManifestList`](https://github.com/apache/iceberg/blob/95a578d7ad96aa3c82dccd305f556c83b3360528/core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java#L262) rewrites the manifest list, it copies the contained manifests and only updates their paths. [Manifest lists are rewritten first and manifest files are rewritten second](https://github.com/apache/iceberg/blob/main/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java#L289-L297), so the utility has no way of knowing final sizes of the rewritten manifest files. This is very similar to https://github.com/apache/iceberg/issues/12554.

Mismatch between manifest size in manifest list and size of the physical manifest file may cause correctness problems in some query engines. For example if the actual manifest file is larger than its size recorded in manifest lists, Trino may only read part of it and ignore the rest, effectively skipping manifest entries (referenced data and delete files) silently.

This is reproducible in Iceberg 1.8.1 at least, and based on the source code this is also true for later versions.

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start with RewriteTablePathUtil.rewriteManifestList and the manifest rewrite ordering in RewriteTablePathSparkAction.java. Trace how rewritten manifest paths and physical sizes are produced, then verify that manifest-list sizes match the resulting manifest files and that the reported skipped-entry scenario is prevented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.