apache / apache/seatunnel

[Feature][Connector-V2] The csv, text, json,xml,parquet and orc file types of the file type connector (localfile, hdfs, s3) as source support sharding

Open
#10,129 0 comments 3 reactions 1 assignee Claimed by @chl-wxp View on GitHub
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 13h
Merged PRs (30d)
203

Description

### Search before asking

- [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.

### Description

When encountering large files or very large files to read, the file is divided according to a certain number of bytes, and parallel reading can improve reading efficiency.

### Usage Scenario

_No response_

### Implementation scope
| | text | csv | json | xml | parquet | orc |
| -------------- | ---- | ---- | ---- | ---- | -------- | ---- |
| local file | done | done | done | | done | doing |
| hdfs | done | done | done | | done | |
| s3 | | | | | | |
| obs | | | | | | |
| oss | | | | | | |
### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.