apache / apache/iceberg

`rewrite_table_path` throws FileNotFoundException on DELETED position delete manifest entries

Open Beginner friendly
#16,662 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.11.0 (latest release)

### Query engine

Spark

### Please describe the bug 🐞

## Environment
- Apache Iceberg: 1.11.0
- Spark: 3.5
- Format version: v2 (position deletes)

## Description

`rewrite_table_path` throws `FileNotFoundException` on position delete `.parquet`
files when the table has had `rewrite_position_delete_files` (or any operation
that deletes position delete files) followed by `expire_snapshots` run on it.

## Steps to Reproduce

1. Create a v2 table with position delete files.
2. Run `rewrite_position_delete_files` — this creates a new snapshot whose
manifest contains DELETED entries for the old position delete files and
ADDED entries for the new ones.
3. Run `expire_snapshots` — this garbage-collects the old position delete
`.parquet` files that are no longer referenced by any live snapshot.
4. Run `rewrite_table_path` (full rewrite, no `start_version`).

## Expected Behavior

`rewrite_table_path` completes successfully, rewriting only live position
delete files.

## Actual Behavior

`rewrite_table_path` throws `FileNotFoundException` (wrapped in
`UncheckedIOException`) on the position delete `.parquet` files that were
garbage-collected in step 3.

## Root Cause

In `RewriteTablePathUtil.writeDeleteFileEntry`, the `POSITION_DELETES` case
unconditionally adds the file to `result.toRewrite()` regardless of the
manifest entry's status

```java
// core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java
case POSITION_DELETES:
DeleteFile posDeleteFile = newPositionDeleteEntry(file, spec, sourcePrefix, targetPrefix);
appendEntryWithFile(entry, writer, posDeleteFile);
if (entry.isLive() && snapshotIds.contains(entry.snapshotId())) {
result.copyPlan().add(...);
}
result.toRewrite().add(file.copy()); // <-- unconditional: adds DELETED entries too
return result;

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start in core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java, focusing on writeDeleteFileEntry and the POSITION_DELETES branch, then trace the rewrite_table_path handling of manifest entry statuses. Reproduce the sequence with position deletes, rewrite_position_delete_files, expire_snapshots, and rewrite_table_path; done means live files are rewritten without attempting to open garbage-collected deleted files.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.