[Umbrella][FIP-27] Remove Mandatory System Columns From Fluss Lake Tables
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
Umbrella issue for [FIP-27: Remove Mandatory System Columns From Fluss Lake Tables](https://cwiki.apache.org/confluence/display/FLUSS/FIP-27%3A+Remove+Mandatory+System+Columns+From+Fluss+Lake+Tables).
New Fluss lake tables should use a clean physical schema containing only user-defined columns by default. Existing lake tables containing `__bucket`, `__offset`, and `__timestamp` must remain readable and writable without schema migration.
#### Sub-issues
- [x] #3902
- [x] #3903
- [ ] #3904
- [x] #3905
#### Overall requirements
- Stop adding the three mandatory Fluss system columns to newly created lake tables.
- Stop imposing system-metadata-based physical ordering on newly created lake tables.
- Detect the physical schema of existing lake tables and preserve their legacy or clean layout.
- Make tiering writers and lake readers support both layouts.
- Allow data lake tiering to be disabled and re-enabled without requiring a schema migration.
- Keep system columns disabled by default. Any future explicit opt-in mechanism is outside this FIP.
#### Compatibility requirements
- New lake storage plugins and Flink connectors must continue reading legacy tables.
- New tiering services must continue writing the legacy layout when the target table contains the system columns.
- Old tiering services must not process newly created clean lake tables.
- Old Flink connectors using `FULL` startup mode must not read newly created clean lake tables.
- The safe upgrade order is lake-reading Flink connectors and lake storage plugins, tiering service, and then the Fluss cluster.
### Willingness to contribute
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.