Need Support for Schema Evolution (Table Structure Changes)
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Motivation
Problem Description
When developing a Kafka Connect Sink Connector for Fluss, we need Schema Evolution (table structure changes). Currently, Fluss 0.8.0-incubating's TableChange interface only supports SetOption and ResetOption, and does not support schema change operations such as adding columns (AddColumn) or dropping columns (DropColumn).
Use Case
The Kafka Connect Sink Connector needs to:
Automatically detect upstream schema changes (new fields)
Automatically add corresponding columns to Fluss tables
Avoid data write failures due to schema mismatches
Current Limitations
After reviewing the source code, we found:
The org.apache.fluss.metadata.TableChange interface only defines:
SetOption - Set table options
ResetOption - Reset table options
The Admin.alterTable() method accepts List, but the server-side implementation (CoordinatorService.java) throws InvalidAlterTableException when encountering non-option type changes
Test code (FlussAdminITCase.java) only tests table option modifications, not column-related operations
Expected Functionality
We hope the TableChange interface can support the following Schema Evolution operations:
AddColumn - Add new columns (support specifying data type, nullable, default value, etc.)
DropColumn - Drop columns (optional: support data migration strategies)
ModifyColumn - Modify column properties (data type, nullability, etc.)
RenameColumn - Rename columns
Similar to the Schema Evolution functionality provided by Flink's TableChange interface.
Impact
Cannot automatically handle schema changes in Kafka Connect
Requires manual intervention for table structure modifications
Affects automation and operational efficiency
Environment Information
Fluss Version: 0.8.0-incubating
Modules Used: fluss-client, fluss-common
Development Scenario: Kafka Connect Sink Connector
Related Code Locations
TableChange interface: fluss-common/src/main/java/org/apache/fluss/metadata/TableChange.java
Admin.alterTable method: fluss-client/src/main/java/org/apache/fluss/client/admin/Admin.java:260
Server-side implementation: fluss-server/src/main/java/org/apache/fluss/server/coordinator/CoordinatorService.java:340-363
Thank you very much!
### Solution
_No response_
### Anything else?
_No response_
### Willingness to contribute
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading fluss-common/src/main/java/org/apache/fluss/metadata/TableChange.java, the Admin.alterTable method in fluss-client/src/main/java/org/apache/fluss/client/admin/Admin.java, and the CoordinatorService handling at the cited lines. Run the existing FlussAdminITCase tests; done requires a decided scope and coverage for the requested schema-change operations without the server rejecting them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100