apache / apache/iceberg

RemoveDanglingDeletes does not remove dangling v2 file-scoped position delete files (only DVs are handled)

Open
#17,203 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

main (development)

### Query engine

Spark

### Please describe the bug 🐞

`RemoveDanglingDeletesSparkAction` cleans up dangling deletion vectors (v3, PUFFIN) but does **not** remove dangling **v2 file-scoped position-delete files** (PARQUET). A file-scoped position delete whose referenced data file has been removed, but whose sequence number is `>=` the partition's minimum data sequence number, is never cleaned up and lingers in table metadata indefinitely.

**Root cause (on `main`):**

- `findDanglingDvs()` performs the correct referenced-data-file left-join, but restricts it to PUFFIN:

```java
// spark/.../actions/RemoveDanglingDeletesSparkAction.java
Dataset dvs =
loadMetadataTable(table, MetadataTableType.DELETE_FILES)
.where(col("file_format").equalTo(FileFormat.PUFFIN.name())); // PUFFIN only
```

v2 PARQUET position deletes are therefore excluded from the referenced-file check.

- `findDanglingDeletes()` only removes position deletes whose sequence number is *below* the partition minimum:

```java
col("data_file.content").equalTo("1")
.and(col("sequence_number").$less(col("min_data_sequence_number")))
```

A file-scoped position delete whose referenced data file is gone, but whose `sequence_number >= min_data_sequence_number` (because another live data file in the same partition keeps the minimum low), is not caught by either method.

**Result:** such position-delete files are unremovable via `remove_dangling_deletes`, bloating table metadata and adding scan-planning overhead.

**Steps to reproduce:**

1. Format-version-2, partitioned table. Append two data files `A` and `B` to the same partition (sequence 1).
2. Add a file-scoped position-delete file referencing only `A` (sequence 2).
3. Remove/overwrite `A` while `B` remains (sequence 3). The partition minimum data sequence number stays 1 (from `B`).
4. Run `remove_dangling_deletes`. Observe: the position-delete file referencing the now-absent `A` is **not** removed (its sequence 2 >= partition min 1, and it is not PUFFIN).

**Expected behavior:** a file-scoped position-delete file whose only referenced data file is absent should be detected as dangling and removed, the same way dangling DVs are.

**Proposed fix (I can contribute):** generalize the referenced-data-file left-join used for DVs to also cover v2 file-scoped position deletes - deriving the referenced data file from the delete file's `lower_bounds`/`upper_bounds` for the `file_path` field when they are equal - while guarding so that only single-data-file (file-scoped) deletes are flagged, never partition-scoped deletes that may still cover live data. I would include a reproducer test.

### Willingness to contribute

- [X] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start in spark/.../actions/RemoveDanglingDeletesSparkAction.java, reading findDanglingDvs() and findDanglingDeletes() to understand their existing metadata joins and sequence filtering. Reproduce the format-version-2 partitioned-table case described in the issue, then add a test covering a file-scoped PARQUET position delete whose referenced data file is absent. Done means remove_dangling_deletes removes that delete without flagging partition-scoped deletes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
67/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.