iceberg实际写入数量会比numWrite数量少
- Dominant language
- Java
- Stars
- 4.1k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar issues.
### What happened
MySQL读取200W数据写入iceberg累加器numRead和numWrite都为200w,iceberg实际查询只有60W
### What you expected to happen
MySQL读取200W数据写入iceberg累加器numRead和numWrite都为200w,iceberg实际查询只有60W
### How to reproduce
{
"job": {
"content": [{
"reader": {
"name": "mysqlreader",
"parameter": {
"column": [{
"name": "id",
"type": "bigint"
}, {
"name": "update_time",
"type": "datetime"
}
],
"connection": [{
"jdbcUrl": ["jdbc:mysql://192.168.20.233:3306/test_sink"],
"schema": "test_sink",
"table": ["test_sink"]
}
],
"customSql": "SELECT id,update_time FROM test_sink.test_sink",
"password": "root",
"username": "root"
}
},
"writer": {
"name": "icebergwriter",
"parameter": {
"column": [{
"name": "ID",
"type": "VARCHAR"
}, {
"name": "value",
"type": "VARCHAR"
}
],
"defaultFS": "hdfs://192.168.42.131:9000",
"hadoopConfig": {
"dfs.client.datanode-restart.timeout": "30",
"dfs.namenode.decommission.interval": "30",
"fs.hdfs.impl.disable.cache": "true"
},
"path": "hdfs://192.168.42.131:9000//flink/warehouse/20221230//icedb/TB_B_JOB_INFO2/",
"writeMode": "overwrite"
}
}
}
],
"setting": {
"speed": {
"bytes": 0,
"channel": 1
}
}
}
}
### Anything else
_No response_
### Version
1.12_release
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Assessment
This issue has not been assessed yet.