apache / apache/datafusion

Consolidate SessionConfig and ConfigOptions

Open
#3,887 3 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

While messing with thishttps://github.com/apache/arrow-datafusion/pull/3885 I found it quite confusing that `SessionConfig` is different than `ConfigOptions` and they had different ways of serializing / deserializing their values.

The new `ConfigOptions` was added by @andygrove https://github.com/apache/arrow-datafusion/pull/2754 and we can now see and set them via environment variables etc and are documented 🎉

https://github.com/apache/arrow-datafusion/blob/b654fdea697b9aec1cb487e292dd288e5c9d09e3/datafusion/core/src/config.rs#L235-L239

But it seems like `SessionConfig` is the old way and is still used (e.g. a field was just added by @Dandandan in https://github.com/apache/arrow-datafusion/pull/3846)

**Describe the solution you'd like**
It would be nice to unify the two (by moving all SessionConfig values into `ConfigOptions`)

https://github.com/apache/arrow-datafusion/blob/b654fdea697b9aec1cb487e292dd288e5c9d09e3/datafusion/core/src/execution/context.rs#L1128-L1156

Which is what I believe this comment in to_props is talking about:
https://github.com/apache/arrow-datafusion/blob/b654fdea697b9aec1cb487e292dd288e5c9d09e3/datafusion/core/src/execution/context.rs#L1282-L1289

**Describe alternatives you've considered**
Leave split brained as is

**Additional context**
https://github.com/apache/arrow-datafusion/issues/3821

Contributor guide

Open the contributing guide

Research direction

Start with datafusion/core/src/config.rs, especially the ConfigOptions definition and its serialization paths, then read the SessionConfig uses in datafusion/core/src/execution/context.rs around lines 1128-1156 and 1282-1289. Trace how both configuration types are read, set, and serialized. Done means the SessionConfig values are unified into ConfigOptions without retaining the split behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.