apache / apache/fluss

[Umbrella][FIP-27] Remove Mandatory System Columns From Fluss Lake Tables

Open
#2,411 3 comments 1 reaction 1 assignee Claimed by @fhan688 View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

Umbrella issue for [FIP-27: Remove Mandatory System Columns From Fluss Lake Tables](https://cwiki.apache.org/confluence/display/FLUSS/FIP-27%3A+Remove+Mandatory+System+Columns+From+Fluss+Lake+Tables).

New Fluss lake tables should use a clean physical schema containing only user-defined columns by default. Existing lake tables containing `__bucket`, `__offset`, and `__timestamp` must remain readable and writable without schema migration.

#### Sub-issues

- [x] #3902
- [x] #3903
- [ ] #3904
- [x] #3905

#### Overall requirements

- Stop adding the three mandatory Fluss system columns to newly created lake tables.
- Stop imposing system-metadata-based physical ordering on newly created lake tables.
- Detect the physical schema of existing lake tables and preserve their legacy or clean layout.
- Make tiering writers and lake readers support both layouts.
- Allow data lake tiering to be disabled and re-enabled without requiring a schema migration.
- Keep system columns disabled by default. Any future explicit opt-in mechanism is outside this FIP.

#### Compatibility requirements

- New lake storage plugins and Flink connectors must continue reading legacy tables.
- New tiering services must continue writing the legacy layout when the target table contains the system columns.
- Old tiering services must not process newly created clean lake tables.
- Old Flink connectors using `FULL` startup mode must not read newly created clean lake tables.
- The safe upgrade order is lake-reading Flink connectors and lake storage plugins, tiering service, and then the Fluss cluster.

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.