DTStack / DTStack/chunjun

flinkx1.12性能问题:mysql-hive 如何提高离线同步效率

Open
#1,138 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
Java
Stars
4.1k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

### Search before asking

- [ ] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar question.

- [ ] I had googled my question but i didn't get any help.

- [ ] I had read the documentation: [ChunJun doc](https://dtstack.github.io/chunjun) but it didn't help me.

### Description

1、数据量9000万+时,mysql同步至hive时,一是不能设置多通道,因为hive sink组件不支持 二是、需要设置fetchSize 实现可读,我设置2048000,但是同步依然很慢,大概需要4个小时
2、j机器性能很好,mysql数据库服务器也很好,我想问如何提交同步效率
![1660180537(1)](https://user-images.githubusercontent.com/34857750/184048940-b397da28-7b7c-4e3e-a47f-c0f37d12863d.png)

![1660180613(1)](https://user-images.githubusercontent.com/34857750/184049000-5f6cd4c0-8fa1-4851-83d9-a9345a47cd6b.png)

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.