apache / apache/arrow-rs-object-store
Update objects to be easier to change without breaking API releases
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Since object_store is "low" in the dependency stack, if we make a breaking change to the API, we then need to wait for that breaking change releases in all other downstream dependencies
So for example, it can take up to 4 months to get into DataFusion (wait 3 months for next major arrow release and then a month for the next major DataFusion release)
That timeframe means that any changes that require non breaking API changes take substantial time to arrive downstream.
Thus allowing new features to be added without breaking downstream changes is more valuable
For example, because [`PutOpts`](https://docs.rs/object_store/latest/object_store/struct.PutOptions.html) is a `pub` struct with `pub` fields, we can not add any new fields without a breaking API release
https://github.com/apache/arrow-rs-object-store/blob/0661843e130b201c803a26cf6e3dd0a197c06d1e/src/lib.rs#L1619-L1638
- This came up as part of https://github.com/apache/arrow-rs-object-store/pull/548#discussion_r2550140973
@crepererum added a `CopyOptions` to reduce replication in the API and to set us up for
- https://github.com/apache/arrow-rs-object-store/issues/297
However, we realized during review that adding cross bucked copy support would likely require adding a new fields to `CopyOptions` and thus another breaking API change.
**Describe the solution you'd like**
Some way that we can add to the existing options without breaking API changes
**Describe alternatives you've considered**
@tustvold suggests: https://github.com/apache/arrow-rs-object-store/pull/548#discussion_r2551290103 following the [`Attributes`](https://docs.rs/object_store/latest/object_store/struct.Attributes.html) model: https://github.com/apache/arrow-rs-object-store/blob/0661843e130b201c803a26cf6e3dd0a197c06d1e/src/attributes.rs#L25-L60
> I think the only way to achieve this would be to make the options enumerable, i.e. a list of some non_exhaustive enum, that way implementations can error if they encounter an option they don't recognise.
>
> Effectively this would take the pattern of Attributes and apply it more broadly.
>
> We could do this, but I think that's probably a separate ticket, and we'd want to do it globally.
**Additional context**
Contributor guide
Research direction
Start with the PutOptions definition in src/lib.rs at the linked lines, then read src/attributes.rs and the discussion on pull request 548. Compare the existing options with the Attributes model and determine what globally applicable approach would allow future options without breaking downstream APIs; done means a concrete design with its compatibility implications agreed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100