apache / apache/gravitino

[FEATURE] Add builtin-iceberg-rewrite-manifests maintenance job

Open
#11,196 3 comments 0 reactions 0 assignees View on GitHub
2.0.0 feature
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### Describe the feature

Add a new built-in Iceberg maintenance job `builtin-iceberg-rewrite-manifests` that rewrites Iceberg manifest files to optimize scan planning performance. This would use Iceberg's `RewriteManifests` action (via Spark's `rewrite_manifests` procedure) to consolidate small manifests and rewrite manifests with improved partition specs.

### Motivation

Over time, Iceberg tables accumulate many small manifest files from frequent commits. This slows down scan planning because the engine must open and read each manifest to determine which data files match a query filter. Rewriting manifests into fewer, larger files with better-aligned partition specs improves query planning performance significantly. The existing `builtin-iceberg-rewrite-data-files` job addresses data file layout but not manifest file bloat.

PR #10500 (merged) added Trino-side delegation for `rewrite_manifests` as a procedure, but there is no server-side built-in job for automatic policy-driven manifest optimization.

### Describe the solution

- Create `IcebergRewriteManifestsJob` in `maintenance/jobs/src/main/java/org/apache/gravitino/maintenance/jobs/iceberg/` following the same pattern as `IcebergRewriteDataFilesJob`
- Register it in `BuiltInJobTemplateProvider` as `builtin-iceberg-rewrite-manifests`
- Support configurable parameters: `use_caching` (boolean)
- Consider adding manifest-related metrics (e.g., manifest count, avg manifest size) to `IcebergUpdateStatsAndMetricsJob` for policy-driven triggering

### Additional context

Related: #10280 (closed: Trino-side procedure delegation), #8864 (umbrella: add built-in jobs for table maintenance)

Contributor guide

Open the contributing guide

Research direction

Start with IcebergRewriteDataFilesJob in maintenance/jobs/src/main/java/org/apache/gravitino/maintenance/jobs/iceberg/ and compare its structure with the requested manifest action. Then inspect BuiltInJobTemplateProvider to locate job registration, and IcebergUpdateStatsAndMetricsJob for the optional metrics context. Done means the new built-in job is registered as builtin-iceberg-rewrite-manifests and supports the use_caching parameter.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.