apache / apache/doris-flink-connector

[Bug] yarn-session模式,无法同步数据,但没有报错异常

Open
#324 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
385
Forks
279
Avg merge
2d 2h
Merged PRs (30d)
6

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.

### Version

java version "1.8.0_212"
flink-1.17.1
hadoop3.3.4
下面三个jar都试过了
-rw-rw-r--. 1 liuqh liuqh 8452171 2月 23 17:28 flink-doris-connector-1.17-1.4.0.jar
-rw-rw-r--. 1 liuqh liuqh 11915843 2月 23 17:28 flink-doris-connector-1.17-1.5.0.jar
-rw-rw-r--. 1 liuqh liuqh 12193974 2月 23 17:28 flink-doris-connector-1.17-1.5.1.jar

### What's Wrong?

在Standalone与Yarn Application模式下都能正常运行。但在Yarn Session模式下,数据同步存在问题。例如,当MySQL库中的test表有5条数据时,开启整库同步后,虽然成功创建了test表到Doris,但doris表数据为0条。随后对MySQL中的test表结构修改,表结构能够更新到Doris,但doris的test数据依然是0条

### What You Expected?

成功同步数据

### How to Reproduce?

`
## 1.5.0yarn-session没办法删除数据,但是可以同步表结构
bin/flink run \
-Dexecution.checkpointing.interval=10s \
-Dstate.checkpoints.dir=hdfs://hadoop102:8020/flink/checkpoints/report_sync \
-Dstate.savepoints.dir=hdfs://hadoop102:8020/flink/flink-savepoints/report_sync \
-Dparallelism.default=1 \
-c org.apache.doris.flink.tools.cdc.CdcTools \
job_jar/flink-doris-connector-1.17-1.5.0.jar \
mysql-sync-database \
--database test \
--mysql-conf \
server-id=5310-5313 \
--mysql-conf hostname=192.168.100.128 \
--mysql-conf port=3306 \
--mysql-conf username=root \
--mysql-conf password='123456' \
--mysql-conf database-name=test \
--sink-conf fenodes=192.168.88.120:8030 \
--sink-conf username=root \
--sink-conf password='Ryuu1234@' \
--sink-conf jdbc-url=jdbc:mysql://192.168.88.120:9030 \
--sink-conf sink.label-prefix=testx_sync2 \
--table-conf replication_num=1 \
--including-tables "test|test_tb01"
`

### Anything Else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start with the provided CdcTools mysql-sync-database command and reproduce it in Yarn Session mode using the listed Flink, Hadoop, and connector versions. Compare it with Standalone and Yarn Application runs; done means the existing MySQL rows are synchronized to Doris as well as subsequent schema changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, mysql
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.