apache / apache/iceberg

rewrite_table_path fails with NotFoundException on expired position delete files referenced by DELETED manifest entries

Open
#17,498 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.11.0 (latest release)

### Query engine

Spark

### Please describe the bug 🐞

`rewrite_table_path` permanently fails with `NotFoundException` / `NoSuchKey` on tables where a position delete file (or deletion vector) was dropped by compaction and later physically removed by `expire_snapshots`. The table itself is perfectly healthy — no reader ever touches the removed file — but it can no longer be replicated, either with a full rewrite or with an incremental rewrite whose delta spans the expiry.

#### Root cause

`RewriteTablePathUtil.writeDeleteFileEntry` applies a filter when building the copy plan: only entries that are live and added within the requested delta are copied. Entries that are `DELETED` (history markers) or outside the delta are still written into the rewritten manifest, but correctly excluded from the copy plan.

However, the physical-rewrite set does not apply the same filter — every position delete entry is queued for rewriting unconditionally:

```java
if (entry.isLive() && snapshotIds.contains(entry.snapshotId())) {
result.copyPlan().add(...);
}
result.toRewrite().add(file.copy()); // unconditional
```

Position delete files must be physically rewritten (they embed absolute data file paths), so `RewriteTablePathSparkAction` opens every file in `toRewrite`. When the entry is a `DELETED` marker whose underlying file was removed by `expire_snapshots`, the open fails and the whole procedure aborts — even though the file was never going to be copied.

Because the `DELETED` entry is baked into the current snapshot's delete manifest, the failure is not transient: every subsequent `rewrite_table_path` run fails the same way.

#### Reproduction (format version 3)

1. Create a table, append data files.
2. Commit a row delta with a deletion vector (`newRowDelta().addDeletes(dv)`).
3. Run `rewrite_data_files` — compaction merges the deletes and drops the DV, leaving a `DELETED` entry in the current snapshot's delete manifest.
4. Run `expire_snapshots` expiring everything older than the compaction snapshot — the DV file is now physically removed from storage, while the `DELETED` entry referencing it survives.
5. Run `rewrite_table_path` (full rewrite) — fails with `NotFoundException` from `PuffinReader` while opening the removed DV.

Note the `DELETED` entry actually passes the `snapshotIds.contains(entry.snapshotId())` check (it is stamped with the compaction snapshot's id), so it is only the `isLive()` half of the copy-plan guard that excludes it from copying — while the rewrite set ignores both.

#### Incremental variant

1. Full `rewrite_table_path` to a target (run 1).
2. On the source: compaction drops a position delete file, then `expire_snapshots` removes it from storage.
3. Incremental `rewrite_table_path` with a `startVersion`/`endVersion` spanning the expiry — the delta manifests still carry the entry (as `DELETED`, or as `EXISTING` outside the delta), it is queued for physical rewriting, and the run fails.

This breaks the common setup of periodic incremental replication combined with routine table maintenance on the source.

#### Non-fatal variant (format version 2)

In v2, compaction leaves the dropped position delete file on storage, so nothing fails — but every dead entry's file is opened, fully rewritten, and written to staging, then never copied (it is not in the copy plan). Wasted I/O and staging storage proportional to the table's history of dropped delete files.

#### Expected behavior

The physical-rewrite set should be aligned with the copy plan: an entry that is excluded from the copy plan is kept in the rewritten manifest exactly as the source has it and must not be opened. This is already how the data-file and equality-delete branches behave — only the position-delete branch is inconsistent.

The `main` branch is not affected: `positionDeletesToRewrite()` collects files to rewrite from live manifest entries. This affects the released line, where the set is built inside `writeDeleteFileEntry`.

### Willingness to contribute

- [x] I can contribute a fix for this bug independently

Contributor guide

Open the contributing guide

Research direction

Start at RewriteTablePathUtil.writeDeleteFileEntry and compare its position-delete handling with RewriteTablePathSparkAction and positionDeletesToRewrite(). Reproduce the format-version-3 scenario with rewrite_table_path and confirm that expired DELETED entries are not opened; done means full and incremental rewrites complete without NotFoundException while preserving the manifest entry.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.