apache / apache/gravitino

[FEATURE] Add builtin-iceberg-remove-orphan-files maintenance job

Closed
#11,195 0 comments 0 reactions 0 assignees View on GitHub
2.0.0 feature
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 17h
Merged PRs (30d)
339

Description

### Describe the feature

Add a new built-in Iceberg maintenance job `builtin-iceberg-remove-orphan-files` that identifies and removes orphaned data and metadata files from Iceberg table storage locations. This would use Iceberg's `RemoveOrphanFiles` action (via Spark's `remove_orphan_files` procedure) to reclaim storage from files that are no longer referenced by any table snapshot.

### Motivation

Orphan files accumulate from failed writes, incomplete transactions, schema evolution, or concurrent operations. Without periodic cleanup, these files waste significant storage, especially in high-write-volume environments. The existing maintenance jobs don't address orphan file cleanup.

PR #10500 (merged) added Trino-side delegation for `remove_orphan_files` as a procedure, but there is no server-side built-in job for automatic policy-driven cleanup.

### Describe the solution

- Create `IcebergRemoveOrphanFilesJob` in `maintenance/jobs/src/main/java/org/apache/gravitino/maintenance/jobs/iceberg/` following the same pattern as `IcebergRewriteDataFilesJob`
- Register it in `BuiltInJobTemplateProvider` as `builtin-iceberg-remove-orphan-files`
- Support configurable parameters: `older_than` (timestamp — default 3 days), `location` (custom location override), `dry_run` (preview-only mode)
- Optionally add a corresponding policy type for scheduled orphan cleanup

### Additional context

Related: #10280 (closed: Trino-side procedure delegation), #8864 (umbrella: add built-in jobs for table maintenance)

Contributor guide

Open the contributing guide

Research direction

Start with maintenance/jobs/src/main/java/org/apache/gravitino/maintenance/jobs/iceberg/IcebergRewriteDataFilesJob and BuiltInJobTemplateProvider to follow the existing job and registration patterns. The work is done when builtin-iceberg-remove-orphan-files is registered and supports older_than, location, and dry_run, with any policy addition resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.