apache / apache/arrow-rs-object-store

Refactoring the configuration system

Open
#719 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Each per-store config key in object_store is currently duplicated across ~8 hand-maintained locations:
- the XxxConfigKey variant,
- its doc + "Supported keys:" list,
- the builder field,
- the AsRef arm,
- the FromStr arm (with all aliases),
- the with_config write arm,
- the get_config_value read arm,
- and the with_ convenience setter.

That is a lot of manual sync points which drift independently.
Keeping it in sync by review alone is not realistic if we are honest.

**Describe the solution you'd like**

A small internal proc-macro derive that makes the struct field the single source of truth:
```rust
#[derive(ObjectStoreConfig)]
#[object_store(config_key = ClientConfigKey, error_path = crate::Error::UnknownConfigurationKey, error_store =
"HTTP")]
pub struct ClientOptions {
/// Set timeout for the overall request.
timeout: Option>,
#[config(setter = skip)] // hand-written setter clears http2_only too
http1_only: ConfigValue,
#[config(skip)]
secrete_internal_type: (),
// ... etc
}
```

The code noted above is auto-generated, so the user does not need to deal with this.

**Describe alternatives you've considered**

I considered
- a `macro_rules!` proc-macro, but i could not quite make the API make the same "level of sense" as the derive-macro one.
- DataFusion's `config_namespace!` approach which seems overkill. It pulls in a ConfigField trait + visitor infrastructure for the nested-namespace case we don't have...

This comes with the downside of needing another crate, not sure if this is a deal breaker.

**Additional context**

I think a good path for such a thing is to do it in an iterative way, first starting with `ClientOptions` and many tests to ensure that everything works as intendet.

Contributor guide

Open the contributing guide

Research direction

Start by locating ClientOptions and the existing per-store configuration key handling. Review the current generated-equivalent paths and design the derive macro incrementally for ClientOptions, including the documented skip and setter behavior. Add many tests to confirm the generated configuration behavior stays synchronized and works as intended.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.