apache / apache/doris-flink-connector
[Bug] 多表同步运行一段时间cdc刷写日志正常打印 doris中相关无数据
- Dominant language
- Java
- Stars
- 385
- Forks
- 279
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 6
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Version
1.6.1
### What's Wrong?
前一天下午以 shell命令的形式启动 启动参数:
mysql-sync-database --database xxx--mysql-conf hostname=xxx --mysql-conf port=3306 --mysql-conf username=xxx--mysql-conf password=xxx --mysql-conf database-name=xxx--mysql-conf scan.startup.mode=latest-offset --mysql-conf server-id=5400-5410 --sink-conf fenodes=doris001:8030 --sink-conf username=xxx--sink-conf password=xxx--sink-conf jdbc-url=jdbc:mysql://xxx:9030 --sink-conf sink.label-prefix=label --table-conf replication_num=3 --table-conf enable_unique_key_merge_on_write=true --including-tables "table01|table02|table03" --table-suffix _full
今天早上查看数据,业务库mysql已经更新到8点多的数据 但是doris中数据最近的是在凌晨3点,查看taskmanager日志 正常打印刷写数据到doris信息,但是doris库查不到该时间段相关数据。又过一会查看taskmanager数据,业务库有了新数据,taskmanager也不打印刷写数据记录信息了,整个过程checkpoint都是正常执行,日志里也没有报错。。。
之前还遇到过不同步数据的情况:taskmanager打印列的信息 信息中的数据都是 内存地址信息 数据也不同步,重启才可以。
### What You Expected?
是不是参数不对?或者写多了?
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start with the mysql-sync-database shell invocation and compare MySQL data with Doris while reviewing TaskManager logs and checkpoints. Investigate the reported period where flush messages continue but Doris has no recent rows, followed by missing flush logs and the earlier memory-address output. Done means the cause is identified and the connector reliably synchronizes the affected tables or documents the required configuration and reproduction steps.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100