apache / apache/doris-flink-connector

[Bug] 多表同步运行一段时间cdc刷写日志正常打印 doris中相关无数据

Open
#453 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
385
Forks
279
Avg merge
2d 2h
Merged PRs (30d)
6

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.

### Version

1.6.1

### What's Wrong?

前一天下午以 shell命令的形式启动 启动参数:
mysql-sync-database --database xxx--mysql-conf hostname=xxx --mysql-conf port=3306 --mysql-conf username=xxx--mysql-conf password=xxx --mysql-conf database-name=xxx--mysql-conf scan.startup.mode=latest-offset --mysql-conf server-id=5400-5410 --sink-conf fenodes=doris001:8030 --sink-conf username=xxx--sink-conf password=xxx--sink-conf jdbc-url=jdbc:mysql://xxx:9030 --sink-conf sink.label-prefix=label --table-conf replication_num=3 --table-conf enable_unique_key_merge_on_write=true --including-tables "table01|table02|table03" --table-suffix _full
今天早上查看数据,业务库mysql已经更新到8点多的数据 但是doris中数据最近的是在凌晨3点,查看taskmanager日志 正常打印刷写数据到doris信息,但是doris库查不到该时间段相关数据。又过一会查看taskmanager数据,业务库有了新数据,taskmanager也不打印刷写数据记录信息了,整个过程checkpoint都是正常执行,日志里也没有报错。。。
之前还遇到过不同步数据的情况:taskmanager打印列的信息 信息中的数据都是 内存地址信息 数据也不同步,重启才可以。

### What You Expected?

是不是参数不对?或者写多了?

### How to Reproduce?

_No response_

### Anything Else?

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start with the mysql-sync-database shell invocation and compare MySQL data with Doris while reviewing TaskManager logs and checkpoints. Investigate the reported period where flush messages continue but Doris has no recent rows, followed by missing flush logs and the earlier memory-address output. Done means the cause is identified and the connector reliably synchronizes the affected tables or documents the required configuration and reproduction steps.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.