[Bug] [connectors-sftp] When using SFTP for synchronization, if there are a large number of files in the directory, performance issues will occur.
- Dominant language
- Java
- Stars
- 9.7k
- Forks
- 2.4k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 204
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
在文件同步的 connectors 中,有个查询文件的方法org.apache.seatunnel.connectors.seatunnel.file.source.reader.AbstractReadStrategy.getFileNamesByPath ,
FileStatus[] stats = hadoopFileSystemProxy.listStatus(path);
有两个问题
1. 目录中有大批量文件时, 会用到 getWorkingDirectory 方法, 这个方法,每次都会创建一个connect ,然后 获取 homeDir, 是否可以使用类变量的方式,毕竟 根目录不会改变,profiler 分析如下
测试的同步作业有 4000 个文件, 记录时间内,92 的开销都在这里。
2. 文件过滤,如果有 file_filter_pattern 配置,应该将这个过滤前置到最开始的位置,避免更大的内存开销。
### SeaTunnel Version
2.3.11
### SeaTunnel Config
```conf
ignore
```
### Running Command
```shell
ignore
```
### Error Exception
```log
none
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.