DTStack / DTStack/chunjun

从mysql 到mysql,无法进行断点续传

Open
#1,478 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
Java
Stars
4.1k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar question.

- [X] I had googled my question but i didn't get any help.

- [X] I had read the documentation: [ChunJun doc](https://dtstack.github.io/chunjun) but it didn't help me.

### Description

目前要进行mysql 到mysql 的离线同步任务,想验证断点续传功能
这是我的json:
id是自增字段

> {
> "job": {
> "content": [{
> "reader": {
> "parameter": {
> "username": "root",
> "password": "123456",
> "connection": [{
> "jdbcUrl": ["jdbc:mysql://node-156:3306/isdms?useUnicode=true&characterEncoding=utf8"],
> "table": ["flink_reader_table"]
> }],
> "column": [{
> "name": "id",
> "type": "int"
> },
> {
> "name": "name",
> "type": "string"
> },
> {
> "name": "age",
> "type": "string"
> },
> {
> "name": "bz",
> "type": "string"
> }],
> "queryTimeOut": 1000,
> "requestAccumulatorInterval": 2
> },
> "name": "mysqlreader"
> },
> "writer": {
> "name": "mysqlwriter",
> "parameter": {
> "username": "root",
> "password": "123456",
> "connection": [
> {
> "jdbcUrl": "jdbc:mysql://node-156:3306/isdms?useSSL=false",
> "table": ["flink_writer_table"]
> }
> ],
> "writeMode": "insert",
> "column": ["id","name","age","bz"],
> "batchSize": 1024
> }
> }
> }],
> "setting": {
> "speed": {
> "bytes": 0,
> "channel": 1
> },
> "restore": {
> "isStream": false,
> "isRestore": true,
> "restoreColumnName": "id",
> "restoreColumnIndex": 0
> },
> "errorLimit" : {
> "record": 100,
> "percentage": 10.0
> }
> }
> }
> }

flink 配置文件:
![image](https://user-images.githubusercontent.com/15731406/211308531-94b6b864-2dc2-4a13-a94b-98bbcb274b47.png)

运行脚本:

> bash /u01/wm/flink/chunjun1.12.7/chunjun/bin/chunjun-yarn-session.sh \
> -job /u01/wm/flink/chunjun1.12.7/chunjun/wm_test/oracle-mysql-ddxc1.json \
> -confProp \{\"yarn.application.id\":\"application_1672202306355_0985\",\"execution.checkpointing.interval\":\"20000\"\
> \"execution.checkpointing.externalized-checkpoint-retention\":\"RETAIN_ON_CANCELLATION\"\
> \"execution.checkpoint.mode\":\"EXACTLY_ONCE\"\}

中断任务后,可看到,100万条数据,目前只同步了21万
![image](https://user-images.githubusercontent.com/15731406/211308591-a2647fd1-615e-4fe6-b21f-f9c4313302b1.png)

在界面中找到checkpoint记录的中断位置
![image](https://user-images.githubusercontent.com/15731406/211308617-476ee50c-e731-4670-ac07-a600a50e4f15.png)

在运行命令的confProp中加上 hdfs://......,再次运行

> bash /u01/wm/flink/chunjun1.12.7/chunjun/bin/chunjun-yarn-session.sh \
> -job /u01/wm/flink/chunjun1.12.7/chunjun/wm_test/oracle-mysql-ddxc1.json \
> -confProp \{\"yarn.application.id\":\"application_1672202306355_0985\",\
> \"execution.checkpointing.interval\":\"20000\",\
> \"execution.savepoint.path\":\"hdfs://hdp26/flinkx/checkpoints/matedata/a5ac89ffb89dba8cd7c5af8d6875b02f/chk-4\"\
> \"execution.checkpointing.externalized-checkpoint-retention\":\"RETAIN_ON_CANCELLATION\"\
> \"execution.checkpoint.mode\":\"EXACTLY_ONCE\"\}

结果并没有从上次断的地方继续传,而是从头开始传
不知道哪里的配置有问题
各位大佬能帮忙看下吗?

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.