apache / apache/iceberg

Flink: Add RemoveDanglingDeleteFiles support to the Flink maintenance API

Open
#16,138 1 comment 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

# Problem

The `RemoveDanglingDeleteFiles` interface is defined in `iceberg-api` but the only
implementation is `RemoveDanglingDeletesSparkAction` (Spark-only). Flink users have
no equivalent, which causes equality delete files to accumulate indefinitely in
tables managed by the Flink maintenance API.

### Root cause

`RewriteDataFiles` in the Flink maintenance pipeline does not remove dangling delete
files after compaction. There are two existing cleanup paths in the Spark
implementation:

**Path A — `ManifestFilterManager` (unreliable)**
During a `RewriteFiles` commit, `ManifestFilterManager` removes delete files whose
sequence number is less than the global minimum data file sequence number across the
entire table. In practice, this condition is rarely satisfied: data files that have
already reached the target size are never rewritten, keeping their original low
sequence numbers and permanently anchoring the global minimum below the eq-delete
sequence numbers.

**Path B — `remove-dangling-deletes=true` (Spark-only, #6126)**
`RewriteDataFilesSparkAction` offers a `remove-dangling-deletes` option that invokes
`RemoveDanglingDeletesSparkAction` after compaction. It correctly computes
`min(data_file.sequence_number)` per partition using a Spark DataFrame join on the
`ENTRIES` metadata table, then calls `table.newRewrite().deleteFile(danglingDelete).commit()`.
This is the correct fix for Path A's shortcomings.

The Flink maintenance API has neither path working reliably.

### Observed impact

In a partitioned V2 table written by `IcebergSink` (equality deletes):

- After multiple rounds of `RewriteDataFiles`, data files that have already reached
the target size are excluded from rewrite groups and retain their original low
sequence numbers.
- These files anchor the global `minDataSequenceNumber`, preventing Path A from
ever removing the eq-delete files.
- `removedEqualityDeleteFiles` is always `null` in every commit summary.
- Equality delete file count grows monotonically with each write, never decreasing.

The only workaround is to temporarily set `rewriteAll(true)`, which forces all data
files — including large ones — into rewrite groups, draining the global minimum and
allowing Path A to fire. This is wasteful and should not be the normal operating mode.

### Summary

Is there an existing mechanism in the Flink maintenance API that handles this case,
or would it make sense to add `RemoveDanglingDeleteFiles` support here?

### Query engine

Flink

### Willingness to contribute

- [ ] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start with the Flink maintenance API's RewriteDataFiles pipeline and compare its cleanup behavior with RewriteDataFilesSparkAction and RemoveDanglingDeletesSparkAction. Trace how RemoveDanglingDeleteFiles is defined in iceberg-api and how the Spark action computes per-partition data-file sequence numbers. Done means Flink compaction removes dangling equality delete files without requiring rewriteAll(true), with coverage for the reported partitioned-table case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.