apache / apache/arrow-rs-object-store

implement GCS resumable uploads as a multipart upload method

Open
#504 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

When performing a streaming upload to GCS, the only option in this crate is [Multipart Uploads](https://cloud.google.com/storage/docs/multipart-uploads). This works the same way it does for S3, and has the same limitations: you cannot perform an upload with >10000 parts. This presents a challenge for large uploads with unknown size. You have to pick a part size large enough to allow the largest plausible upload size, but still small enough that your buffers don't grow too large, and your cost of retrying a part upload is not too great.

The [Resumable upload](https://cloud.google.com/storage/docs/resumable-uploads) API does not have this limitation, you can upload as many parts as you want at whatever size you want.

The downside of resumable uploads is that part uploads must be done serially, you cannot upload parts in parallel or out-of-order.

**Describe the solution you'd like**

I'd like the option to use resumable uploads when using `put_multipart`. I would configure this in the client builder.

**Describe alternatives you've considered**
I can't think of one.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by tracing the client builder and the existing `put_multipart` implementation in the object_store crate. Read the linked GCS multipart and resumable-upload documentation, then determine how a builder option can select serial resumable uploads while preserving the existing multipart behavior. Done means the option is exposed, resumable uploads work for unknown-length streams, and the relevant behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, rust
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.