从mysql 到mysql,无法进行断点续传
- Dominant language
- Java
- Stars
- 4.1k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar question.
- [X] I had googled my question but i didn't get any help.
- [X] I had read the documentation: [ChunJun doc](https://dtstack.github.io/chunjun) but it didn't help me.
### Description
目前要进行mysql 到mysql 的离线同步任务,想验证断点续传功能
这是我的json:
id是自增字段
> {
> "job": {
> "content": [{
> "reader": {
> "parameter": {
> "username": "root",
> "password": "123456",
> "connection": [{
> "jdbcUrl": ["jdbc:mysql://node-156:3306/isdms?useUnicode=true&characterEncoding=utf8"],
> "table": ["flink_reader_table"]
> }],
> "column": [{
> "name": "id",
> "type": "int"
> },
> {
> "name": "name",
> "type": "string"
> },
> {
> "name": "age",
> "type": "string"
> },
> {
> "name": "bz",
> "type": "string"
> }],
> "queryTimeOut": 1000,
> "requestAccumulatorInterval": 2
> },
> "name": "mysqlreader"
> },
> "writer": {
> "name": "mysqlwriter",
> "parameter": {
> "username": "root",
> "password": "123456",
> "connection": [
> {
> "jdbcUrl": "jdbc:mysql://node-156:3306/isdms?useSSL=false",
> "table": ["flink_writer_table"]
> }
> ],
> "writeMode": "insert",
> "column": ["id","name","age","bz"],
> "batchSize": 1024
> }
> }
> }],
> "setting": {
> "speed": {
> "bytes": 0,
> "channel": 1
> },
> "restore": {
> "isStream": false,
> "isRestore": true,
> "restoreColumnName": "id",
> "restoreColumnIndex": 0
> },
> "errorLimit" : {
> "record": 100,
> "percentage": 10.0
> }
> }
> }
> }
flink 配置文件:

运行脚本:
> bash /u01/wm/flink/chunjun1.12.7/chunjun/bin/chunjun-yarn-session.sh \
> -job /u01/wm/flink/chunjun1.12.7/chunjun/wm_test/oracle-mysql-ddxc1.json \
> -confProp \{\"yarn.application.id\":\"application_1672202306355_0985\",\"execution.checkpointing.interval\":\"20000\"\
> \"execution.checkpointing.externalized-checkpoint-retention\":\"RETAIN_ON_CANCELLATION\"\
> \"execution.checkpoint.mode\":\"EXACTLY_ONCE\"\}
中断任务后,可看到,100万条数据,目前只同步了21万

在界面中找到checkpoint记录的中断位置

在运行命令的confProp中加上 hdfs://......,再次运行
> bash /u01/wm/flink/chunjun1.12.7/chunjun/bin/chunjun-yarn-session.sh \
> -job /u01/wm/flink/chunjun1.12.7/chunjun/wm_test/oracle-mysql-ddxc1.json \
> -confProp \{\"yarn.application.id\":\"application_1672202306355_0985\",\
> \"execution.checkpointing.interval\":\"20000\",\
> \"execution.savepoint.path\":\"hdfs://hdp26/flinkx/checkpoints/matedata/a5ac89ffb89dba8cd7c5af8d6875b02f/chk-4\"\
> \"execution.checkpointing.externalized-checkpoint-retention\":\"RETAIN_ON_CANCELLATION\"\
> \"execution.checkpoint.mode\":\"EXACTLY_ONCE\"\}
结果并没有从上次断的地方继续传,而是从头开始传
不知道哪里的配置有问题
各位大佬能帮忙看下吗?
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Assessment
This issue has not been assessed yet.