apache / apache/doris-flink-connector
[Bug] yarn-session模式,无法同步数据,但没有报错异常
- Dominant language
- Java
- Stars
- 385
- Forks
- 279
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 6
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Version
java version "1.8.0_212"
flink-1.17.1
hadoop3.3.4
下面三个jar都试过了
-rw-rw-r--. 1 liuqh liuqh 8452171 2月 23 17:28 flink-doris-connector-1.17-1.4.0.jar
-rw-rw-r--. 1 liuqh liuqh 11915843 2月 23 17:28 flink-doris-connector-1.17-1.5.0.jar
-rw-rw-r--. 1 liuqh liuqh 12193974 2月 23 17:28 flink-doris-connector-1.17-1.5.1.jar
### What's Wrong?
在Standalone与Yarn Application模式下都能正常运行。但在Yarn Session模式下,数据同步存在问题。例如,当MySQL库中的test表有5条数据时,开启整库同步后,虽然成功创建了test表到Doris,但doris表数据为0条。随后对MySQL中的test表结构修改,表结构能够更新到Doris,但doris的test数据依然是0条
### What You Expected?
成功同步数据
### How to Reproduce?
`
## 1.5.0yarn-session没办法删除数据,但是可以同步表结构
bin/flink run \
-Dexecution.checkpointing.interval=10s \
-Dstate.checkpoints.dir=hdfs://hadoop102:8020/flink/checkpoints/report_sync \
-Dstate.savepoints.dir=hdfs://hadoop102:8020/flink/flink-savepoints/report_sync \
-Dparallelism.default=1 \
-c org.apache.doris.flink.tools.cdc.CdcTools \
job_jar/flink-doris-connector-1.17-1.5.0.jar \
mysql-sync-database \
--database test \
--mysql-conf \
server-id=5310-5313 \
--mysql-conf hostname=192.168.100.128 \
--mysql-conf port=3306 \
--mysql-conf username=root \
--mysql-conf password='123456' \
--mysql-conf database-name=test \
--sink-conf fenodes=192.168.88.120:8030 \
--sink-conf username=root \
--sink-conf password='Ryuu1234@' \
--sink-conf jdbc-url=jdbc:mysql://192.168.88.120:9030 \
--sink-conf sink.label-prefix=testx_sync2 \
--table-conf replication_num=1 \
--including-tables "test|test_tb01"
`
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start with the provided CdcTools mysql-sync-database command and reproduce it in Yarn Session mode using the listed Flink, Hadoop, and connector versions. Compare it with Standalone and Yarn Application runs; done means the existing MySQL rows are synchronized to Doris as well as subsequent schema changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, mysql
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100