DTStack / DTStack/chunjun

iceberg实际写入数量会比numWrite数量少

Open
#1,496 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
4.1k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar issues.

### What happened

MySQL读取200W数据写入iceberg累加器numRead和numWrite都为200w,iceberg实际查询只有60W

### What you expected to happen

MySQL读取200W数据写入iceberg累加器numRead和numWrite都为200w,iceberg实际查询只有60W

### How to reproduce

{
"job": {
"content": [{
"reader": {
"name": "mysqlreader",
"parameter": {
"column": [{
"name": "id",
"type": "bigint"
}, {
"name": "update_time",
"type": "datetime"
}
],
"connection": [{
"jdbcUrl": ["jdbc:mysql://192.168.20.233:3306/test_sink"],
"schema": "test_sink",
"table": ["test_sink"]
}
],
"customSql": "SELECT id,update_time FROM test_sink.test_sink",
"password": "root",
"username": "root"
}
},
"writer": {
"name": "icebergwriter",
"parameter": {
"column": [{
"name": "ID",
"type": "VARCHAR"
}, {
"name": "value",
"type": "VARCHAR"
}
],
"defaultFS": "hdfs://192.168.42.131:9000",
"hadoopConfig": {
"dfs.client.datanode-restart.timeout": "30",
"dfs.namenode.decommission.interval": "30",
"fs.hdfs.impl.disable.cache": "true"
},
"path": "hdfs://192.168.42.131:9000//flink/warehouse/20221230//icedb/TB_B_JOB_INFO2/",
"writeMode": "overwrite"
}
}
}
],
"setting": {
"speed": {
"bytes": 0,
"channel": 1
}
}
}
}

### Anything else

_No response_

### Version

1.12_release

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.