apache / apache/seatunnel

[Feature][Connector-V2][File] Add real-time monitoring and reading of file types such as FTP, SFTP, HDFS, local files, etc

Open
#10,414 7 comments 0 reactions 1 assignee Claimed by @yzeng1618 View on GitHub
feature
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

### Search before asking

- [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.

### Description

Add real-time monitoring and reading of file types such as FTP, SFTP, HDFS, local files, etc., and support not to repeatedly transfer files that have already been transferred. Support immediate deletion after transfer, file expiration deletion and time, file backup and deletion after transfer, file scanning interval time, as well as support priority queue, queue size, queue buffer size, thread count and other operations

### Usage Scenario

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.