apache / apache/paimon

[Bug] how can I solve the bias data when sync from mysql to paimon

Open
#1,587 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Paimon version

0.5

### Compute Engine

flink

### Minimal reproduce step

I have met this data bias when sync data from mysql to paimon. and my command is as following, and I have 800000 records in mysql table.

./flink run -Dscan.mode=latest -Dstreaming-read-mode=file -Dexecution.checkpointing.interval=1s -Dclassloader.check-leaked-classloader=false paimon-flink-action-0.5-SNAPSHOT.jar mysql-sync-table --warehouse hdfs://pcosmo-gyos03:8020/user/root/paimon --database paimon_test --table paimon_sync_wangke --computed-columns 'current_login_time=avg(current_login_time)' --mysql-conf hostname=10.206.32.3 --mysql-conf port=9006 --mysql-conf username=root --mysql-conf password=123456 --mysql-conf database-name=paimontest --mysql-conf table-name='wangke' --catalog-conf metastore=hive --catalog-conf uri=thrift://10.206.32.3:9083 --table-conf write-manifest-cache=1024mb --table-conf manifest.full-compaction-threshold-size=16mb --table-conf scan.manifest.parallelism=4 --table-conf scan.split-enumerator.batch-size=4 --table-conf bucket=4 --table-conf changelog-producer=input --table-conf sink.parallelism=4 --table-conf execution.checkpointing.timeout=60min --table-conf execution.checkpointing.interval=5min --table-conf full-compaction.delta-commits=3 --table-conf changelog-producer=full-compaction --table-conf scan.mode=compacted-full --table-conf num-sorted-run.stop-trigger=2147483647 --table-conf sort-spill-threshold=10 --table-conf write-buffer-size=4096mb --table-conf taskmanager.memory.managed.size=1m --table-conf write-buffer-spillable=true

![image](https://github.com/apache/incubator-paimon/assets/8140615/be7cfd6d-5eac-4186-bd2f-8fcd29a5b073)

### What doesn't meet your expectations?

I want to solve this data bias

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Begin by reproducing the supplied Flink mysql-sync-table command and comparing the reported 800,000 MySQL records with the resulting Paimon data. Investigate the computed-column and scan/changelog settings shown in the command; no source file or test is named. Done means identifying the cause of the bias and validating a corrected synchronization result.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.