apache / apache/arrow-rs-object-store

Add source conditions to `ObjectStore::copy_opts`

Open
#667 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

In the process of updating an object in S3 or GCS, I want to first do a buckup of the original object. Currently, the only way to do this consistently when there is potential for concurrent writes, is to first fetch the object with `get_opts`, using an `if_match` condition, and then write it back to the object store. `object_store` has `copy` to do this, but currently it doesn't support conditions on the source. Most object stores (including S3 and GCS) do support this feature.

**Describe the solution you'd like**

Add `if_match` and other supported conditions to `ObjectStore::copy_opts`

**Describe alternatives you've considered**

There is the workaround I mention in the introduction, but it requires streaming the object back and forth from/to the object store.

**Additional context**

The context for this request is the [Icechunk](https://github.com/earth-mover/icechunk/) project, where we use object stores conditionals to implement transactions over large amounts of data.

Contributor guide

Open the contributing guide

Research direction

Start at ObjectStore::copy_opts and compare the existing copy path with get_opts and its if_match condition handling. Check how source conditions are represented across the S3 and GCS object stores, then verify that if_match and other supported conditions are applied to the source copy operation without the streaming workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.