apache / apache/arrow-rs-object-store
[Object Store] provide more utilities for working with object stores
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Many consumers of object-store end up needing to write additional utility functions for validating and building object stores. Most of which is not application specific.
For example, many users have implemented an `Options` struct
- [glaredb](https://github.com/GlareDB/glaredb/blob/e8a1f85af2fb390bcc46dd1cca917651bf8af15e/crates/protogen/src/metastore/types/options.rs#L423)
- [lance](https://github.com/lancedb/lance/blob/main/rust/lance-core/src/io/object_store.rs#L591)
- [delta](https://github.com/delta-io/delta-rs/blob/main/crates/deltalake-core/src/storage/config.rs#L117)
- [polars](https://github.com/pola-rs/polars/blob/55ec54e1626c16a9e3eba7fb63e840ee47b53263/crates/polars-io/src/cloud/options.rs#L48)
Similarly, all of these users have also created utilities for parsing and validating the urls as well
- [glaredb](https://github.com/GlareDB/glaredb/blob/a9c9b460be94f217ddb2283eb2b90a8ae686465e/crates/datasources/src/common/url.rs#L18)
- [lance](https://github.com/lancedb/lance/blob/main/rust/lance-core/src/io/object_store.rs#L934)
- [delta](https://github.com/delta-io/delta-rs/blob/main/crates/deltalake-core/src/storage/config.rs#L58)
- [delta](https://github.com/delta-io/delta-rs/blob/main/crates/deltalake-core/src/table/builder.rs#L405) _(validation)_
- [polars](https://github.com/pola-rs/polars/blob/55ec54e1626c16a9e3eba7fb63e840ee47b53263/crates/polars-io/src/cloud/options.rs#L48)
**Describe the solution you'd like**
move this logic into object_store so users don't need to create the same utilities
**Describe alternatives you've considered**
Each consumer continues to provide their own utility functions/structs.
**Additional context**
Contributor guide
Research direction
Compare the linked Options implementations and URL parsing and validation utilities in GlareDB, Lance, Delta, and Polars to identify the shared requirements. Then inspect the object_store crate to define the public API and scope; the issue provides no target files or tests, so done is not yet specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100