apache / apache/arrow-rs-object-store
implement GCS resumable uploads as a multipart upload method
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
When performing a streaming upload to GCS, the only option in this crate is [Multipart Uploads](https://cloud.google.com/storage/docs/multipart-uploads). This works the same way it does for S3, and has the same limitations: you cannot perform an upload with >10000 parts. This presents a challenge for large uploads with unknown size. You have to pick a part size large enough to allow the largest plausible upload size, but still small enough that your buffers don't grow too large, and your cost of retrying a part upload is not too great.
The [Resumable upload](https://cloud.google.com/storage/docs/resumable-uploads) API does not have this limitation, you can upload as many parts as you want at whatever size you want.
The downside of resumable uploads is that part uploads must be done serially, you cannot upload parts in parallel or out-of-order.
**Describe the solution you'd like**
I'd like the option to use resumable uploads when using `put_multipart`. I would configure this in the client builder.
**Describe alternatives you've considered**
I can't think of one.
**Additional context**
Contributor guide
Research direction
Start by tracing the client builder and the existing `put_multipart` implementation in the object_store crate. Read the linked GCS multipart and resumable-upload documentation, then determine how a builder option can select serial resumable uploads while preserving the existing multipart behavior. Done means the option is exposed, resumable uploads work for unknown-length streams, and the relevant behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, rust
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100