apache / apache/datafusion

chore: Explore combining `SessionConfig` and `RuntimeConfig` into same framework

Open
#23,322 5 comments 0 reactions 0 assignees View on GitHub
core good first issue
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

A followup on #23316

Can we please explore if runtime config can be part of `SessionConfig`? In `SessionConfig` there is already a framework that enforces type checking.

Runtime configs looks looks finite according to `set_runtime_variable`, so should fit into main configs rather than having separate config set

One difference was that the `RuntimeConfig` is potentially shared across multiple `SessionConfigs` (and maybe can also be adjusted during plan execution)

Trying to consolidate `RuntimeConfig` and `SessionConfig` should be considered as an independent project

_Originally posted by @alamb in https://github.com/apache/datafusion/issues/23316#issuecomment-4881517563_

Contributor guide

Open the contributing guide

Research direction

Start by tracing SessionConfig, RuntimeConfig, and set_runtime_variable in the DataFusion source. Compare the existing type-checking framework with runtime configuration, including sharing across multiple SessionConfigs and adjustment during plan execution. Done means documenting whether consolidation is viable and resolving those lifecycle differences in a concrete design.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.