[FEATURE] Add builtin-iceberg-rewrite-manifests maintenance job
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### Describe the feature
Add a new built-in Iceberg maintenance job `builtin-iceberg-rewrite-manifests` that rewrites Iceberg manifest files to optimize scan planning performance. This would use Iceberg's `RewriteManifests` action (via Spark's `rewrite_manifests` procedure) to consolidate small manifests and rewrite manifests with improved partition specs.
### Motivation
Over time, Iceberg tables accumulate many small manifest files from frequent commits. This slows down scan planning because the engine must open and read each manifest to determine which data files match a query filter. Rewriting manifests into fewer, larger files with better-aligned partition specs improves query planning performance significantly. The existing `builtin-iceberg-rewrite-data-files` job addresses data file layout but not manifest file bloat.
PR #10500 (merged) added Trino-side delegation for `rewrite_manifests` as a procedure, but there is no server-side built-in job for automatic policy-driven manifest optimization.
### Describe the solution
- Create `IcebergRewriteManifestsJob` in `maintenance/jobs/src/main/java/org/apache/gravitino/maintenance/jobs/iceberg/` following the same pattern as `IcebergRewriteDataFilesJob`
- Register it in `BuiltInJobTemplateProvider` as `builtin-iceberg-rewrite-manifests`
- Support configurable parameters: `use_caching` (boolean)
- Consider adding manifest-related metrics (e.g., manifest count, avg manifest size) to `IcebergUpdateStatsAndMetricsJob` for policy-driven triggering
### Additional context
Related: #10280 (closed: Trino-side procedure delegation), #8864 (umbrella: add built-in jobs for table maintenance)
Contributor guide
Research direction
Start with IcebergRewriteDataFilesJob in maintenance/jobs/src/main/java/org/apache/gravitino/maintenance/jobs/iceberg/ and compare its structure with the requested manifest action. Then inspect BuiltInJobTemplateProvider to locate job registration, and IcebergUpdateStatsAndMetricsJob for the optional metrics context. Done means the new built-in job is registered as builtin-iceberg-rewrite-manifests and supports the use_caching parameter.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100