apache / apache/arrow-rs-object-store

Support conditional multipart uploads (similar to conditional puts)

Open
#396 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

Support conditional multipart uploads throughout the multipart API, i.e.

- `put_multipart_opts`
- `MultipartUpload`
- `MultipartStore`
- `BufWriter`

## Why?

- Generally useful API to have, as currently, `object_store` cannot easily fulfill use cases that need both a streaming API *and* an optimistic concurrency model easily out of the box
- Provides the opportunity to streamline some APIs (e.g. `PutOptions` and `PutMultipartOpts` only differ on the `mode` field; Supporting `mode` for all multipart operations means they could be unified. The S3 implementation also already uses `CompleteMultipartMode`, but only implements `Create` and `Overwrite` modes)
- Enable concrete use-cases like [this one in polars](https://github.com/pola-rs/polars/issues/23021)

## Caveats

- GCS doesn't support conditional multipart uploads via the currently used XML API


If that sounds like something worth doing, I'd be happy to work on it

Contributor guide

Open the contributing guide

Research direction

Trace conditional put handling into put_multipart_opts, MultipartUpload, MultipartStore, and BufWriter, then compare the existing S3 CompleteMultipartMode support with the GCS XML API limitation. Define how conditional modes propagate through each multipart operation and verify that supported backends preserve optimistic-concurrency behavior without assuming GCS support.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.