[FEATURE] Add Iceberg sort maintenance strategy (end-to-end policy + strategy handler)
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 286
Description
### Describe the feature
Add end-to-end support for a **sort** maintenance strategy for Iceberg tables in the Table Maintenance Service (Optimizer), so a table can be automatically rewritten into a sorted (or z-ordered) data file layout based on a policy — analogous to the existing `system_iceberg_compaction` strategy.
The execution layer already exists: the built-in `builtin-iceberg-rewrite-data-files` job (`IcebergRewriteDataFilesJob`) already supports `--strategy sort` and `--sort-order`. What is missing is the **policy + strategy-recommendation layer** that decides *when* a table should be sorted and emits the corresponding job.
### Motivation
Sorting (and z-ordering) data files significantly improves query performance via better data skipping and file pruning, especially for high-cardinality filter columns. Today the optimizer only has a compaction (`binpack`) strategy (`CompactionStrategyHandler` / `system_iceberg_compaction`); there is no way to express "keep this table sorted by columns X, Y" as a managed policy and have the optimizer schedule it.
This complements the in-flight expire-snapshots (#11194), remove-orphan-files (#11195), and rewrite-manifests (#11196) work under umbrella #8864, reusing the same strategy → job design pattern established in the expire-snapshots design doc (#11206).
### Describe the solution
Following the existing compaction flow and the design pattern in #11206:
1. **Policy content** — add `IcebergSortContent` (similar to `IcebergDataCompactionContent`) with: sort order (list of `{column, direction, nullOrdering}`), sort `strategy` (`sort` | `zorder`), optional `where` filter, and rewrite options (e.g. `target-file-size-bytes`, `min-input-files`).
2. **Policy type** — add `system_iceberg_sort` to `Policy.BuiltInType` and the matching `PolicyContentDTO` / `PolicyDTO` JSON subtypes (mirroring `system_iceberg_compaction`).
3. **Strategy handler** — add `SortStrategyHandler extends BaseExpressionStrategyHandler` (mirroring `CompactionStrategyHandler`), emitting a `JobExecutionContext` that invokes `builtin-iceberg-rewrite-data-files` with `strategy=sort` and the configured `sort-order`. Register it in `GravitinoStrategyProvider`.
4. **Reuse** the existing `IcebergRewriteDataFilesJob` — no new job class is needed.
5. Unit tests for the policy content (serde), the strategy handler (recommendation logic), and the job-options mapping, consistent with the existing compaction / expire-snapshots test coverage.
### Additional context
- Umbrella: #8864 (built-in jobs for table maintenance)
- Sibling strategies: #11194 (expire-snapshots), #11195 (remove-orphan-files), #11196 (rewrite-manifests)
- Design pattern reference: #11206 (expire-snapshots end-to-end design doc)
- Trino-side `rewrite_data_files` (incl. sort) procedure delegation already merged in #10500
I'd like to work on this.
Contributor guide
Assessment
This issue has not been assessed yet.