apache / apache/gravitino

[Improvement] TMS optimizer: support non-identity partition transforms for compaction

Open
#10,358 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### What would you like to be improved?

Current TMS compaction flow is effectively limited to identity partition transforms in stable usage. This blocks practical compaction workflows for common Iceberg partition specs that include non-identity transforms (for example `days(ts)`, `months(ts)`, `years(ts)`, `hours(ts)`, `truncate`, `bucket`) in strategy recommendation and rewrite execution path.

### How should we improve?

- Add end-to-end support for non-identity partition transforms in TMS compaction recommendation and rewrite submission path.
- Ensure generated partition predicates are valid and executable for supported transforms.
- Define clear compatibility behavior for unsupported transforms (explicit fallback or skip policy) and document it.
- Add tests for partition predicate generation and rewrite job integration for major transform types.

Parent Epic: #10360

Contributor guide

Open the contributing guide

Research direction

Start by locating the TMS compaction recommendation and rewrite submission paths, then inspect how identity partition transforms are currently handled. Define the supported-transform behavior and fallback policy from the issue, and add tests for partition predicate generation and rewrite job integration across the major transform types.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.