apache / apache/seatunnel

[Bug] [connectors-sftp] When using SFTP for synchronization, if there are a large number of files in the directory, performance issues will occur.

Open
#9,953 1 comment 0 reactions 1 assignee Claimed by @ZhongLinLeo View on GitHub
improve
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.

### What happened

在文件同步的 connectors 中,有个查询文件的方法org.apache.seatunnel.connectors.seatunnel.file.source.reader.AbstractReadStrategy.getFileNamesByPath ,
FileStatus[] stats = hadoopFileSystemProxy.listStatus(path);
有两个问题
1. 目录中有大批量文件时, 会用到 getWorkingDirectory 方法, 这个方法,每次都会创建一个connect ,然后 获取 homeDir, 是否可以使用类变量的方式,毕竟 根目录不会改变,profiler 分析如下

测试的同步作业有 4000 个文件, 记录时间内,92 的开销都在这里。
2. 文件过滤,如果有 file_filter_pattern 配置,应该将这个过滤前置到最开始的位置,避免更大的内存开销。

### SeaTunnel Version

2.3.11

### SeaTunnel Config

```conf
ignore
```

### Running Command

```shell
ignore
```

### Error Exception

```log
none
```

### Zeta or Flink or Spark Version

_No response_

### Java or Scala Version

_No response_

### Screenshots

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.