RemoveDanglingDeletes does not remove dangling v2 file-scoped position delete files (only DVs are handled)
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
main (development)
### Query engine
Spark
### Please describe the bug 🐞
`RemoveDanglingDeletesSparkAction` cleans up dangling deletion vectors (v3, PUFFIN) but does **not** remove dangling **v2 file-scoped position-delete files** (PARQUET). A file-scoped position delete whose referenced data file has been removed, but whose sequence number is `>=` the partition's minimum data sequence number, is never cleaned up and lingers in table metadata indefinitely.
**Root cause (on `main`):**
- `findDanglingDvs()` performs the correct referenced-data-file left-join, but restricts it to PUFFIN:
```java
// spark/.../actions/RemoveDanglingDeletesSparkAction.java
Dataset dvs =
loadMetadataTable(table, MetadataTableType.DELETE_FILES)
.where(col("file_format").equalTo(FileFormat.PUFFIN.name())); // PUFFIN only
```
v2 PARQUET position deletes are therefore excluded from the referenced-file check.
- `findDanglingDeletes()` only removes position deletes whose sequence number is *below* the partition minimum:
```java
col("data_file.content").equalTo("1")
.and(col("sequence_number").$less(col("min_data_sequence_number")))
```
A file-scoped position delete whose referenced data file is gone, but whose `sequence_number >= min_data_sequence_number` (because another live data file in the same partition keeps the minimum low), is not caught by either method.
**Result:** such position-delete files are unremovable via `remove_dangling_deletes`, bloating table metadata and adding scan-planning overhead.
**Steps to reproduce:**
1. Format-version-2, partitioned table. Append two data files `A` and `B` to the same partition (sequence 1).
2. Add a file-scoped position-delete file referencing only `A` (sequence 2).
3. Remove/overwrite `A` while `B` remains (sequence 3). The partition minimum data sequence number stays 1 (from `B`).
4. Run `remove_dangling_deletes`. Observe: the position-delete file referencing the now-absent `A` is **not** removed (its sequence 2 >= partition min 1, and it is not PUFFIN).
**Expected behavior:** a file-scoped position-delete file whose only referenced data file is absent should be detected as dangling and removed, the same way dangling DVs are.
**Proposed fix (I can contribute):** generalize the referenced-data-file left-join used for DVs to also cover v2 file-scoped position deletes - deriving the referenced data file from the delete file's `lower_bounds`/`upper_bounds` for the `file_path` field when they are equal - while guarding so that only single-data-file (file-scoped) deletes are flagged, never partition-scoped deletes that may still cover live data. I would include a reproducer test.
### Willingness to contribute
- [X] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start in spark/.../actions/RemoveDanglingDeletesSparkAction.java, reading findDanglingDvs() and findDanglingDeletes() to understand their existing metadata joins and sequence filtering. Reproduce the format-version-2 partitioned-table case described in the issue, then add a test covering a file-scoped PARQUET position delete whose referenced data file is absent. Done means remove_dangling_deletes removes that delete without flagging partition-scoped deletes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 67/100