apache / apache/arrow-rs-object-store
Add source conditions to `ObjectStore::copy_opts`
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
In the process of updating an object in S3 or GCS, I want to first do a buckup of the original object. Currently, the only way to do this consistently when there is potential for concurrent writes, is to first fetch the object with `get_opts`, using an `if_match` condition, and then write it back to the object store. `object_store` has `copy` to do this, but currently it doesn't support conditions on the source. Most object stores (including S3 and GCS) do support this feature.
**Describe the solution you'd like**
Add `if_match` and other supported conditions to `ObjectStore::copy_opts`
**Describe alternatives you've considered**
There is the workaround I mention in the introduction, but it requires streaming the object back and forth from/to the object store.
**Additional context**
The context for this request is the [Icechunk](https://github.com/earth-mover/icechunk/) project, where we use object stores conditionals to implement transactions over large amounts of data.
Contributor guide
Research direction
Start at ObjectStore::copy_opts and compare the existing copy path with get_opts and its if_match condition handling. Check how source conditions are represented across the S3 and GCS object stores, then verify that if_match and other supported conditions are applied to the source copy operation without the streaming workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100