apache / apache/arrow-rs-object-store

Update objects to be easier to change without breaking API releases

Open
#553 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Since object_store is "low" in the dependency stack, if we make a breaking change to the API, we then need to wait for that breaking change releases in all other downstream dependencies

So for example, it can take up to 4 months to get into DataFusion (wait 3 months for next major arrow release and then a month for the next major DataFusion release)

That timeframe means that any changes that require non breaking API changes take substantial time to arrive downstream.

Thus allowing new features to be added without breaking downstream changes is more valuable

For example, because [`PutOpts`](https://docs.rs/object_store/latest/object_store/struct.PutOptions.html) is a `pub` struct with `pub` fields, we can not add any new fields without a breaking API release

https://github.com/apache/arrow-rs-object-store/blob/0661843e130b201c803a26cf6e3dd0a197c06d1e/src/lib.rs#L1619-L1638

- This came up as part of https://github.com/apache/arrow-rs-object-store/pull/548#discussion_r2550140973

@crepererum added a `CopyOptions` to reduce replication in the API and to set us up for
- https://github.com/apache/arrow-rs-object-store/issues/297

However, we realized during review that adding cross bucked copy support would likely require adding a new fields to `CopyOptions` and thus another breaking API change.

**Describe the solution you'd like**
Some way that we can add to the existing options without breaking API changes

**Describe alternatives you've considered**
@tustvold suggests: https://github.com/apache/arrow-rs-object-store/pull/548#discussion_r2551290103 following the [`Attributes`](https://docs.rs/object_store/latest/object_store/struct.Attributes.html) model: https://github.com/apache/arrow-rs-object-store/blob/0661843e130b201c803a26cf6e3dd0a197c06d1e/src/attributes.rs#L25-L60

> I think the only way to achieve this would be to make the options enumerable, i.e. a list of some non_exhaustive enum, that way implementations can error if they encounter an option they don't recognise.
>
> Effectively this would take the pattern of Attributes and apply it more broadly.
>
> We could do this, but I think that's probably a separate ticket, and we'd want to do it globally.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start with the PutOptions definition in src/lib.rs at the linked lines, then read src/attributes.rs and the discussion on pull request 548. Compare the existing options with the Attributes model and determine what globally applicable approach would allow future options without breaking downstream APIs; done means a concrete design with its compatibility implications agreed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.