[Improvement] TMS optimizer: support non-identity partition transforms for compaction
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### What would you like to be improved?
Current TMS compaction flow is effectively limited to identity partition transforms in stable usage. This blocks practical compaction workflows for common Iceberg partition specs that include non-identity transforms (for example `days(ts)`, `months(ts)`, `years(ts)`, `hours(ts)`, `truncate`, `bucket`) in strategy recommendation and rewrite execution path.
### How should we improve?
- Add end-to-end support for non-identity partition transforms in TMS compaction recommendation and rewrite submission path.
- Ensure generated partition predicates are valid and executable for supported transforms.
- Define clear compatibility behavior for unsupported transforms (explicit fallback or skip policy) and document it.
- Add tests for partition predicate generation and rewrite job integration for major transform types.
Parent Epic: #10360
Contributor guide
Research direction
Start by locating the TMS compaction recommendation and rewrite submission paths, then inspect how identity partition transforms are currently handled. Define the supported-transform behavior and fallback policy from the issue, and add tests for partition predicate generation and rewrite job integration across the major transform types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100