apache / apache/fluss

Need Support for Schema Evolution (Table Structure Changes)

Open
#2,410 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

Problem Description
When developing a Kafka Connect Sink Connector for Fluss, we need Schema Evolution (table structure changes). Currently, Fluss 0.8.0-incubating's TableChange interface only supports SetOption and ResetOption, and does not support schema change operations such as adding columns (AddColumn) or dropping columns (DropColumn).

Use Case
The Kafka Connect Sink Connector needs to:
Automatically detect upstream schema changes (new fields)
Automatically add corresponding columns to Fluss tables
Avoid data write failures due to schema mismatches

Current Limitations
After reviewing the source code, we found:
The org.apache.fluss.metadata.TableChange interface only defines:
SetOption - Set table options
ResetOption - Reset table options
The Admin.alterTable() method accepts List, but the server-side implementation (CoordinatorService.java) throws InvalidAlterTableException when encountering non-option type changes
Test code (FlussAdminITCase.java) only tests table option modifications, not column-related operations

Expected Functionality
We hope the TableChange interface can support the following Schema Evolution operations:
AddColumn - Add new columns (support specifying data type, nullable, default value, etc.)
DropColumn - Drop columns (optional: support data migration strategies)
ModifyColumn - Modify column properties (data type, nullability, etc.)
RenameColumn - Rename columns
Similar to the Schema Evolution functionality provided by Flink's TableChange interface.

Impact
Cannot automatically handle schema changes in Kafka Connect
Requires manual intervention for table structure modifications
Affects automation and operational efficiency

Environment Information
Fluss Version: 0.8.0-incubating
Modules Used: fluss-client, fluss-common
Development Scenario: Kafka Connect Sink Connector

Related Code Locations
TableChange interface: fluss-common/src/main/java/org/apache/fluss/metadata/TableChange.java
Admin.alterTable method: fluss-client/src/main/java/org/apache/fluss/client/admin/Admin.java:260
Server-side implementation: fluss-server/src/main/java/org/apache/fluss/server/coordinator/CoordinatorService.java:340-363

Thank you very much!

### Solution

_No response_

### Anything else?

_No response_

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading fluss-common/src/main/java/org/apache/fluss/metadata/TableChange.java, the Admin.alterTable method in fluss-client/src/main/java/org/apache/fluss/client/admin/Admin.java, and the CoordinatorService handling at the cited lines. Run the existing FlussAdminITCase tests; done requires a decided scope and coverage for the requested schema-change operations without the server rejecting them.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.