[FEATURE] Add builtin-iceberg-remove-orphan-files maintenance job
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 339
Description
### Describe the feature
Add a new built-in Iceberg maintenance job `builtin-iceberg-remove-orphan-files` that identifies and removes orphaned data and metadata files from Iceberg table storage locations. This would use Iceberg's `RemoveOrphanFiles` action (via Spark's `remove_orphan_files` procedure) to reclaim storage from files that are no longer referenced by any table snapshot.
### Motivation
Orphan files accumulate from failed writes, incomplete transactions, schema evolution, or concurrent operations. Without periodic cleanup, these files waste significant storage, especially in high-write-volume environments. The existing maintenance jobs don't address orphan file cleanup.
PR #10500 (merged) added Trino-side delegation for `remove_orphan_files` as a procedure, but there is no server-side built-in job for automatic policy-driven cleanup.
### Describe the solution
- Create `IcebergRemoveOrphanFilesJob` in `maintenance/jobs/src/main/java/org/apache/gravitino/maintenance/jobs/iceberg/` following the same pattern as `IcebergRewriteDataFilesJob`
- Register it in `BuiltInJobTemplateProvider` as `builtin-iceberg-remove-orphan-files`
- Support configurable parameters: `older_than` (timestamp — default 3 days), `location` (custom location override), `dry_run` (preview-only mode)
- Optionally add a corresponding policy type for scheduled orphan cleanup
### Additional context
Related: #10280 (closed: Trino-side procedure delegation), #8864 (umbrella: add built-in jobs for table maintenance)
Contributor guide
Research direction
Start with maintenance/jobs/src/main/java/org/apache/gravitino/maintenance/jobs/iceberg/IcebergRewriteDataFilesJob and BuiltInJobTemplateProvider to follow the existing job and registration patterns. The work is done when builtin-iceberg-remove-orphan-files is registered and supports older_than, location, and dry_run, with any policy addition resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100