apache / apache/paimon

[Bug] The addition of a new field in MySQL CDC may result in data loss during the data synchronization process

Open
#2,864 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Paimon version

0.7

### Compute Engine

flink 1.17.2

### Minimal reproduce step

When the paimon table is set to write-only mode, if there are changes to the table structure and the replace method in StoreSinkWriteImpl is executed, the checkpoint obtained from restoreFiles is an empty array. This results in the sequenceNumbers property of the newly created MergeTreeWriter being set to 0.

### What doesn't meet your expectations?

The addition of a new field in MySQL CDC may result in data loss during the data

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with StoreSinkWriteImpl, focusing on the replace method and the restoreFiles checkpoint behavior described in the report. Trace how an empty checkpoint affects MergeTreeWriter sequenceNumbers during MySQL CDC schema changes, then verify that adding a field does not lose synchronized data.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.