[Bug] how can I solve the bias data when sync from mysql to paimon
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Paimon version
0.5
### Compute Engine
flink
### Minimal reproduce step
I have met this data bias when sync data from mysql to paimon. and my command is as following, and I have 800000 records in mysql table.
./flink run -Dscan.mode=latest -Dstreaming-read-mode=file -Dexecution.checkpointing.interval=1s -Dclassloader.check-leaked-classloader=false paimon-flink-action-0.5-SNAPSHOT.jar mysql-sync-table --warehouse hdfs://pcosmo-gyos03:8020/user/root/paimon --database paimon_test --table paimon_sync_wangke --computed-columns 'current_login_time=avg(current_login_time)' --mysql-conf hostname=10.206.32.3 --mysql-conf port=9006 --mysql-conf username=root --mysql-conf password=123456 --mysql-conf database-name=paimontest --mysql-conf table-name='wangke' --catalog-conf metastore=hive --catalog-conf uri=thrift://10.206.32.3:9083 --table-conf write-manifest-cache=1024mb --table-conf manifest.full-compaction-threshold-size=16mb --table-conf scan.manifest.parallelism=4 --table-conf scan.split-enumerator.batch-size=4 --table-conf bucket=4 --table-conf changelog-producer=input --table-conf sink.parallelism=4 --table-conf execution.checkpointing.timeout=60min --table-conf execution.checkpointing.interval=5min --table-conf full-compaction.delta-commits=3 --table-conf changelog-producer=full-compaction --table-conf scan.mode=compacted-full --table-conf num-sorted-run.stop-trigger=2147483647 --table-conf sort-spill-threshold=10 --table-conf write-buffer-size=4096mb --table-conf taskmanager.memory.managed.size=1m --table-conf write-buffer-spillable=true

### What doesn't meet your expectations?
I want to solve this data bias
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Begin by reproducing the supplied Flink mysql-sync-table command and comparing the reported 800,000 MySQL records with the resulting Paimon data. Investigate the computed-column and scan/changelog settings shown in the command; no source file or test is named. Done means identifying the cause of the bias and validating a corrected synchronization result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100