Core: Orphaned merged DV file on cache invalidation and commit failure
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
## Background
Commit `de4101118` (PR #15006) introduced duplicate DV detection and merging in `MergingSnapshotProducer.mergeDVs()`. Each time `mergeDVs()` is invoked, it writes a new Puffin file (e.g. `merged-dvs--.puffin`).
The delete manifests produced by `newDeleteFilesAsManifests()` are cached. When the cache is invalidated (a subsequent `addDeletes` call sets `hasNewDeleteFiles=true`), the code only deletes the cached **manifest** files but leaves the **Puffin** file produced by the previous `mergeDVs()` call orphaned on disk.
The same orphan occurs on **commit failure**: `cleanUncommittedAppends` deletes uncommitted delete manifests but never deletes the Puffin files they referenced.
## Reproducer
```
RowDelta.addDeletes(dv1, dv2) → apply()
// writes merged-dvs-...-1.puffin, caches delete manifests
→ addDeletes(dv3)
// invalidates cache: deletes manifest, but puffin file is orphaned
→ commit()
```
## Impact
Resource leak — orphaned Puffin files accumulate in the table's data directory and can only be removed by `remove_orphan_files`. Does not corrupt committed data; read correctness is preserved.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start with MergingSnapshotProducer.mergeDVs() and newDeleteFilesAsManifests(), then trace cache invalidation after addDeletes and cleanUncommittedAppends on commit failure. Confirm that both paths clean up the Puffin files referenced by discarded delete manifests, and verify the reproducer no longer leaves merged-dvs-*.puffin files orphaned.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100