apache / apache/fluss

[FIP-27] Support clean and legacy Hudi lake table schemas

Open
#3,904 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

Implement the Hudi part of [FIP-27](https://cwiki.apache.org/confluence/display/FLUSS/FIP-27%3A+Remove+Mandatory+System+Columns+From+Fluss+Lake+Tables).

Parent issue: #2411

New Hudi lake tables should contain only user-defined columns. Existing tables containing `__bucket`, `__offset`, and `__timestamp` must remain readable and writable. Hudi metadata columns must continue to be handled independently from the legacy Fluss system columns.

#### Scope

- Create new Hudi tables without the mandatory Fluss system columns.
- Detect whether an existing Hudi table uses the legacy or clean Fluss layout.
- Make Hudi tiering writers and regular and sorted readers support both layouts.
- Preserve the existing layout when data lake tiering is disabled and re-enabled.
- Keep Hudi metadata-column projection independent from Fluss system-column projection.
- Reject partial or type-incompatible legacy system-column layouts with a clear error.

#### Acceptance criteria

- A newly created Hudi lake table exposes only user-defined columns in addition to Hudi-managed metadata.
- Existing legacy Hudi tables continue to be readable and writable without migration.
- Disabling and re-enabling tiering preserves the existing physical layout.
- Tests cover clean and legacy layouts, writer paths, regular and sorted readers, projections, and re-enabling tiering.
- Tests cover DFS and HMS catalog modes and log and primary-key tables where supported.

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with FIP-27 and parent issue #2411, then trace the Hudi tiering writers and regular and sorted readers for DFS and HMS catalog modes. Done means clean tables omit Fluss system columns, legacy tables remain readable and writable, projections stay independent, re-enabling tiering preserves the layout, and tests cover the listed layouts and table types.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.