[Feature] Support implicit/hidden partitions for Fluss tables
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
discussion thread https://lists.apache.org/thread/p657bmg1n5k1fllxhxg0zdg9kcls9w9p
### Motivation
### Background
Currently Fluss partitioning requires partition keys to be declared as physical table columns. For primary-key tables, the partition key also has to be a subset of the primary key. On the write path, the partition value is extracted directly from each row.
This means users need to materialize partition columns such as `dt`, `hh`, or `event_day` in the table schema and in every written row, even when the partition value can be derived from an existing field such as `event_time` or from ingestion time.
### Problem
For ingestion and lakehouse-style use cases, this leaks storage layout into the logical table schema and makes pipelines more cumbersome:
- Upstream sources may not contain the desired partition column.
- Users have to add derived columns only for partition routing.
- The derived partition column can become inconsistent with the source timestamp column.
- Primary-key table definitions are harder because the partition column must be included in the primary key.
- It is different from lake formats such as Iceberg, where hidden partitions and partition transforms are common.
Fluss already supports auto partition creation, but auto partitioning still depends on an explicit partition column in the table schema. It does not provide implicit or hidden partitioning.
### Solution
Add support for implicit / hidden partitions, or partition transforms, so the partition value can be derived without exposing an extra physical column in the table schema.
A minimal first step could support time-based transforms on one existing timestamp/date column, then extend to multi-level transforms later.
### Anything else?
This probably needs a design/FIP because it touches table metadata, validation, client write routing, catalog integration, and partition pruning semantics.
Related current behavior in the codebase:
- `TableDescriptor` validates partition keys against schema/primary-key columns.
- `PartitionGetter` extracts partition values from row fields.
- Auto partitioning only creates partitions automatically; it does not make partition columns implicit.
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with TableDescriptor validation and PartitionGetter, then trace the auto-partitioning, catalog, write-routing, and partition-pruning paths named in the issue. Because the issue calls for a design/FIP, first establish the semantics and scope for hidden time-based transforms, including metadata, validation, routing, and pruning; done means an accepted design with an implementation plan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100