apache / apache/arrow-rs-object-store
Enable AWS client to copy objects >5GB in size
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I'm trying to copy objects within a bucket as part of my data ingestion workflow. However, when thse objects exceed 5GB in size, this fails with an error. There's a built-in limit in the S3 API for the maximum size of a copy operation.
**Describe the solution you'd like**
I've outlined the solution i'd prefer in #561 - essentially the copy method should detect large copies and transparently switch to a multipart workflow.
**Describe alternatives you've considered**
Alternatively, we could also expose this API as part of the multipart-upload API.
**Additional context**
Contributor guide
Research direction
Start by reading the existing copy method and the multipart-upload API, then review the approach outlined in issue #561. The work is done when copies over the S3 5GB limit transparently use a multipart workflow, or the API is exposed through multipart upload as decided by the implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100