Flink: Add RemoveDanglingDeleteFiles support to the Flink maintenance API
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
# Problem
The `RemoveDanglingDeleteFiles` interface is defined in `iceberg-api` but the only
implementation is `RemoveDanglingDeletesSparkAction` (Spark-only). Flink users have
no equivalent, which causes equality delete files to accumulate indefinitely in
tables managed by the Flink maintenance API.
### Root cause
`RewriteDataFiles` in the Flink maintenance pipeline does not remove dangling delete
files after compaction. There are two existing cleanup paths in the Spark
implementation:
**Path A — `ManifestFilterManager` (unreliable)**
During a `RewriteFiles` commit, `ManifestFilterManager` removes delete files whose
sequence number is less than the global minimum data file sequence number across the
entire table. In practice, this condition is rarely satisfied: data files that have
already reached the target size are never rewritten, keeping their original low
sequence numbers and permanently anchoring the global minimum below the eq-delete
sequence numbers.
**Path B — `remove-dangling-deletes=true` (Spark-only, #6126)**
`RewriteDataFilesSparkAction` offers a `remove-dangling-deletes` option that invokes
`RemoveDanglingDeletesSparkAction` after compaction. It correctly computes
`min(data_file.sequence_number)` per partition using a Spark DataFrame join on the
`ENTRIES` metadata table, then calls `table.newRewrite().deleteFile(danglingDelete).commit()`.
This is the correct fix for Path A's shortcomings.
The Flink maintenance API has neither path working reliably.
### Observed impact
In a partitioned V2 table written by `IcebergSink` (equality deletes):
- After multiple rounds of `RewriteDataFiles`, data files that have already reached
the target size are excluded from rewrite groups and retain their original low
sequence numbers.
- These files anchor the global `minDataSequenceNumber`, preventing Path A from
ever removing the eq-delete files.
- `removedEqualityDeleteFiles` is always `null` in every commit summary.
- Equality delete file count grows monotonically with each write, never decreasing.
The only workaround is to temporarily set `rewriteAll(true)`, which forces all data
files — including large ones — into rewrite groups, draining the global minimum and
allowing Path A to fire. This is wasteful and should not be the normal operating mode.
### Summary
Is there an existing mechanism in the Flink maintenance API that handles this case,
or would it make sense to add `RemoveDanglingDeleteFiles` support here?
### Query engine
Flink
### Willingness to contribute
- [ ] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Start with the Flink maintenance API's RewriteDataFiles pipeline and compare its cleanup behavior with RewriteDataFilesSparkAction and RemoveDanglingDeletesSparkAction. Trace how RemoveDanglingDeleteFiles is defined in iceberg-api and how the Spark action computes per-partition data-file sequence numbers. Done means Flink compaction removes dangling equality delete files without requiring rewriteAll(true), with coverage for the reported partitioned-table case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100