[Feature][chunjun-connector-hdfs] hdfs消费百亿数据时的速度不如spark
- Dominant language
- Java
- Stars
- 4.1k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar feature requirement.
### Description
hdfs到hdfs的同步任务 ：数据源有300亿，300个文件，设定并行度为300，任务完成时间为3个小时左右，而spark运行相同的任务仅50多分钟。通过分析chunjun-core 与chunjun-connector-hdfs 代码 猜测是因为metric指标（包括脏数据）处理影响了消费速度，通过摘除chunjun-core中metric指标监控（不包含numWrite），hdfs的同步速度快了三倍。具体的，将去除指标chunjun-core与chunjun-connector-hdfs合并到一起创建了新的hdfs连接器。
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Assessment
This issue has not been assessed yet.