apache / apache/gravitino

[FEATURE] Add Iceberg sort maintenance strategy (end-to-end policy + strategy handler)

Open
#11,699 0 comments 0 reactions 1 assignee Claimed by @bharos View on GitHub
2.0.0 feature
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 17h
Merged PRs (30d)
286

Description

### Describe the feature

Add end-to-end support for a **sort** maintenance strategy for Iceberg tables in the Table Maintenance Service (Optimizer), so a table can be automatically rewritten into a sorted (or z-ordered) data file layout based on a policy — analogous to the existing `system_iceberg_compaction` strategy.

The execution layer already exists: the built-in `builtin-iceberg-rewrite-data-files` job (`IcebergRewriteDataFilesJob`) already supports `--strategy sort` and `--sort-order`. What is missing is the **policy + strategy-recommendation layer** that decides *when* a table should be sorted and emits the corresponding job.

### Motivation

Sorting (and z-ordering) data files significantly improves query performance via better data skipping and file pruning, especially for high-cardinality filter columns. Today the optimizer only has a compaction (`binpack`) strategy (`CompactionStrategyHandler` / `system_iceberg_compaction`); there is no way to express "keep this table sorted by columns X, Y" as a managed policy and have the optimizer schedule it.

This complements the in-flight expire-snapshots (#11194), remove-orphan-files (#11195), and rewrite-manifests (#11196) work under umbrella #8864, reusing the same strategy → job design pattern established in the expire-snapshots design doc (#11206).

### Describe the solution

Following the existing compaction flow and the design pattern in #11206:

1. **Policy content** — add `IcebergSortContent` (similar to `IcebergDataCompactionContent`) with: sort order (list of `{column, direction, nullOrdering}`), sort `strategy` (`sort` | `zorder`), optional `where` filter, and rewrite options (e.g. `target-file-size-bytes`, `min-input-files`).
2. **Policy type** — add `system_iceberg_sort` to `Policy.BuiltInType` and the matching `PolicyContentDTO` / `PolicyDTO` JSON subtypes (mirroring `system_iceberg_compaction`).
3. **Strategy handler** — add `SortStrategyHandler extends BaseExpressionStrategyHandler` (mirroring `CompactionStrategyHandler`), emitting a `JobExecutionContext` that invokes `builtin-iceberg-rewrite-data-files` with `strategy=sort` and the configured `sort-order`. Register it in `GravitinoStrategyProvider`.
4. **Reuse** the existing `IcebergRewriteDataFilesJob` — no new job class is needed.
5. Unit tests for the policy content (serde), the strategy handler (recommendation logic), and the job-options mapping, consistent with the existing compaction / expire-snapshots test coverage.

### Additional context

- Umbrella: #8864 (built-in jobs for table maintenance)
- Sibling strategies: #11194 (expire-snapshots), #11195 (remove-orphan-files), #11196 (rewrite-manifests)
- Design pattern reference: #11206 (expire-snapshots end-to-end design doc)
- Trino-side `rewrite_data_files` (incl. sort) procedure delegation already merged in #10500

I'd like to work on this.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.