Consolidate SessionConfig and ConfigOptions
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
While messing with thishttps://github.com/apache/arrow-datafusion/pull/3885 I found it quite confusing that `SessionConfig` is different than `ConfigOptions` and they had different ways of serializing / deserializing their values.
The new `ConfigOptions` was added by @andygrove https://github.com/apache/arrow-datafusion/pull/2754 and we can now see and set them via environment variables etc and are documented 🎉
https://github.com/apache/arrow-datafusion/blob/b654fdea697b9aec1cb487e292dd288e5c9d09e3/datafusion/core/src/config.rs#L235-L239
But it seems like `SessionConfig` is the old way and is still used (e.g. a field was just added by @Dandandan in https://github.com/apache/arrow-datafusion/pull/3846)
**Describe the solution you'd like**
It would be nice to unify the two (by moving all SessionConfig values into `ConfigOptions`)
https://github.com/apache/arrow-datafusion/blob/b654fdea697b9aec1cb487e292dd288e5c9d09e3/datafusion/core/src/execution/context.rs#L1128-L1156
Which is what I believe this comment in to_props is talking about:
https://github.com/apache/arrow-datafusion/blob/b654fdea697b9aec1cb487e292dd288e5c9d09e3/datafusion/core/src/execution/context.rs#L1282-L1289
**Describe alternatives you've considered**
Leave split brained as is
**Additional context**
https://github.com/apache/arrow-datafusion/issues/3821
Contributor guide
Research direction
Start with datafusion/core/src/config.rs, especially the ConfigOptions definition and its serialization paths, then read the SessionConfig uses in datafusion/core/src/execution/context.rs around lines 1128-1156 and 1282-1289. Trace how both configuration types are read, set, and serialized. Done means the SessionConfig values are unified into ConfigOptions without retaining the split behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100