从MYSQL同步数据到CLICKHOUSE,速度太慢
- Dominant language
- Java
- Stars
- 4.1k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
### Search before asking
- [ ] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar question.
- [ ] I had googled my question but i didn't get any help.
- [ ] I had read the documentation: [ChunJun doc](https://dtstack.github.io/chunjun) but it didn't help me.
### Description
从MYSQL同步数据到CLICKHOUSE,速度太慢。
版本是2022年11月下的,现在查不到版本号了。MYSQL单表1000多万的数据,同步到ck近30小时。请问要怎么优化速度?
json:
{
"job": {
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"column": [
...
],
"customSql": "xxxxxxxxx",
"fetchSize": 1024,
"where": "GID > '110d4fb092dd44e68d0df3b971e86a95'",
"username": "xxxx",
"password": "xxxx",
"connection": [
{
"jdbcUrl": [
"jdbc:mysql://xxx.xxx.xxxx.xxxx:3306/xxx?serverTimezone=Asia/Shanghai&useUnicode=true&characterEncoding=UTF-8&useSSL=false"
],
"table": [
"xxxxxx"
]
}
]
}
},
"writer": {
"name": "clickhousewriter",
"parameter": {
"username": "default",
"password": "xxxxxx",
"connection": [
{
"jdbcUrl": "jdbc:clickhouse://xxx.xxx.xxx.xxx:8123/xxx",
"table": [
"xxxxx"
]
}
],
"writeMode" : "insert",
"encoding" : "utf-8",
"column": [
...
]
}
}
}
],
"setting": {
"speed": {
"channel": 1,
"bytes": 0
},
"restore": {
"isResore": false
},
"errorLimit" : {
"record": 1
},
"log" : {
"isLogger": true,
"level" : "debug"
}
}
}
}
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Assessment
This issue has not been assessed yet.