rewrite_table_path fails with NotFoundException on expired position delete files referenced by DELETED manifest entries
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
`rewrite_table_path` permanently fails with `NotFoundException` / `NoSuchKey` on tables where a position delete file (or deletion vector) was dropped by compaction and later physically removed by `expire_snapshots`. The table itself is perfectly healthy — no reader ever touches the removed file — but it can no longer be replicated, either with a full rewrite or with an incremental rewrite whose delta spans the expiry.
#### Root cause
`RewriteTablePathUtil.writeDeleteFileEntry` applies a filter when building the copy plan: only entries that are live and added within the requested delta are copied. Entries that are `DELETED` (history markers) or outside the delta are still written into the rewritten manifest, but correctly excluded from the copy plan.
However, the physical-rewrite set does not apply the same filter — every position delete entry is queued for rewriting unconditionally:
```java
if (entry.isLive() && snapshotIds.contains(entry.snapshotId())) {
result.copyPlan().add(...);
}
result.toRewrite().add(file.copy()); // unconditional
```
Position delete files must be physically rewritten (they embed absolute data file paths), so `RewriteTablePathSparkAction` opens every file in `toRewrite`. When the entry is a `DELETED` marker whose underlying file was removed by `expire_snapshots`, the open fails and the whole procedure aborts — even though the file was never going to be copied.
Because the `DELETED` entry is baked into the current snapshot's delete manifest, the failure is not transient: every subsequent `rewrite_table_path` run fails the same way.
#### Reproduction (format version 3)
1. Create a table, append data files.
2. Commit a row delta with a deletion vector (`newRowDelta().addDeletes(dv)`).
3. Run `rewrite_data_files` — compaction merges the deletes and drops the DV, leaving a `DELETED` entry in the current snapshot's delete manifest.
4. Run `expire_snapshots` expiring everything older than the compaction snapshot — the DV file is now physically removed from storage, while the `DELETED` entry referencing it survives.
5. Run `rewrite_table_path` (full rewrite) — fails with `NotFoundException` from `PuffinReader` while opening the removed DV.
Note the `DELETED` entry actually passes the `snapshotIds.contains(entry.snapshotId())` check (it is stamped with the compaction snapshot's id), so it is only the `isLive()` half of the copy-plan guard that excludes it from copying — while the rewrite set ignores both.
#### Incremental variant
1. Full `rewrite_table_path` to a target (run 1).
2. On the source: compaction drops a position delete file, then `expire_snapshots` removes it from storage.
3. Incremental `rewrite_table_path` with a `startVersion`/`endVersion` spanning the expiry — the delta manifests still carry the entry (as `DELETED`, or as `EXISTING` outside the delta), it is queued for physical rewriting, and the run fails.
This breaks the common setup of periodic incremental replication combined with routine table maintenance on the source.
#### Non-fatal variant (format version 2)
In v2, compaction leaves the dropped position delete file on storage, so nothing fails — but every dead entry's file is opened, fully rewritten, and written to staging, then never copied (it is not in the copy plan). Wasted I/O and staging storage proportional to the table's history of dropped delete files.
#### Expected behavior
The physical-rewrite set should be aligned with the copy plan: an entry that is excluded from the copy plan is kept in the rewritten manifest exactly as the source has it and must not be opened. This is already how the data-file and equality-delete branches behave — only the position-delete branch is inconsistent.
The `main` branch is not affected: `positionDeletesToRewrite()` collects files to rewrite from live manifest entries. This affects the released line, where the set is built inside `writeDeleteFileEntry`.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
Contributor guide
Research direction
Start at RewriteTablePathUtil.writeDeleteFileEntry and compare its position-delete handling with RewriteTablePathSparkAction and positionDeletesToRewrite(). Reproduce the format-version-3 scenario with rewrite_table_path and confirm that expired DELETED entries are not opened; done means full and incremental rewrites complete without NotFoundException while preserving the manifest entry.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 74/100