[FIP-27] Support clean and legacy Hudi lake table schemas
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
Implement the Hudi part of [FIP-27](https://cwiki.apache.org/confluence/display/FLUSS/FIP-27%3A+Remove+Mandatory+System+Columns+From+Fluss+Lake+Tables).
Parent issue: #2411
New Hudi lake tables should contain only user-defined columns. Existing tables containing `__bucket`, `__offset`, and `__timestamp` must remain readable and writable. Hudi metadata columns must continue to be handled independently from the legacy Fluss system columns.
#### Scope
- Create new Hudi tables without the mandatory Fluss system columns.
- Detect whether an existing Hudi table uses the legacy or clean Fluss layout.
- Make Hudi tiering writers and regular and sorted readers support both layouts.
- Preserve the existing layout when data lake tiering is disabled and re-enabled.
- Keep Hudi metadata-column projection independent from Fluss system-column projection.
- Reject partial or type-incompatible legacy system-column layouts with a clear error.
#### Acceptance criteria
- A newly created Hudi lake table exposes only user-defined columns in addition to Hudi-managed metadata.
- Existing legacy Hudi tables continue to be readable and writable without migration.
- Disabling and re-enabling tiering preserves the existing physical layout.
- Tests cover clean and legacy layouts, writer paths, regular and sorted readers, projections, and re-enabling tiering.
- Tests cover DFS and HMS catalog modes and log and primary-key tables where supported.
### Willingness to contribute
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with FIP-27 and parent issue #2411, then trace the Hudi tiering writers and regular and sorted readers for DFS and HMS catalog modes. Done means clean tables omit Fluss system columns, legacy tables remain readable and writable, projections stay independent, re-enabling tiering preserves the layout, and tests cover the listed layouts and table types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100