apache / apache/arrow-rs-object-store
Support conditional multipart uploads (similar to conditional puts)
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
Support conditional multipart uploads throughout the multipart API, i.e.
- `put_multipart_opts`
- `MultipartUpload`
- `MultipartStore`
- `BufWriter`
## Why?
- Generally useful API to have, as currently, `object_store` cannot easily fulfill use cases that need both a streaming API *and* an optimistic concurrency model easily out of the box
- Provides the opportunity to streamline some APIs (e.g. `PutOptions` and `PutMultipartOpts` only differ on the `mode` field; Supporting `mode` for all multipart operations means they could be unified. The S3 implementation also already uses `CompleteMultipartMode`, but only implements `Create` and `Overwrite` modes)
- Enable concrete use-cases like [this one in polars](https://github.com/pola-rs/polars/issues/23021)
## Caveats
- GCS doesn't support conditional multipart uploads via the currently used XML API
If that sounds like something worth doing, I'd be happy to work on it
Contributor guide
Research direction
Trace conditional put handling into put_multipart_opts, MultipartUpload, MultipartStore, and BufWriter, then compare the existing S3 CompleteMultipartMode support with the GCS XML API limitation. Define how conditional modes propagate through each multipart operation and verify that supported backends preserve optimistic-concurrency behavior without assuming GCS support.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100