apache / apache/paimon

[Bug] When the paimon-flink-cdc module processes Kafka messages formatted in maxwell-json, if it encounters events lacking the "data" field, such as "table-alter", the program immediately throws an error and enters an unrecoverable state.

Open
#5,092 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

9d2a4605dffc39ddb0e5d0b8733c02bf9ea42c68

### Compute Engine

flink 1.17

### Minimal reproduce step

1.Establish a connection with the MySQL database, Maxwell collector, Kafka, and the paimon-flink-cdc program.
2.Initiate table-level modifications on MySQL tables, specifically table-alter events, such as ALTER TABLE xxx AUTO_INCREMENT = 2300.
3.The paimon-flink-cdc will throw an error message: "invalid maxwell-json format: missing 'data' field."

### What doesn't meet your expectations?

The program encountered an error and cannot be recovered. It is expected to prevent such errors.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in the paimon-flink-cdc module by reproducing the Maxwell-JSON handling with a table-alter event from Kafka. Trace the parsing entry point for messages without a data field; done means the program avoids the unrecoverable error while processing such events.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka, mysql
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.