Minimize the dependency on `SessionState`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
There are many functions in `datafusion-core` that take `SessionState` as arguments but only actually rely on portion of them. This add the additional dependency that is not necessary, therefore blocking us from extracting module out of core #10782.
For example, If we want to pull `CatalogProvider` out of core, we need to pull out `TableProvider` first. But because it has `scan` function that takes `SessionState` which contains `CatalogProviderList` therefore there is a circular dependency. Similar issues are already mentioned in https://github.com/apache/datafusion/issues/11182
### Describe the solution you'd like
I think we need to redesign those functions that take `SessionState` and minimize the dependencies for them.
Given one of the `scan` function here, we can see that we only need `state.config_options().explain` and `state.execution_props()` instead of the whole `SessionState`
https://github.com/apache/datafusion/blob/4bed04e4e312a0b125306944aee94a93c2ff6c4f/datafusion/core/src/datasource/memory.rs#L207-L245
In this case, we can create `TableProviderConext` that encapsulates a subset of the information from `SessionState`.
```rust
pub struct TableProviderConext {
explain_options: ExplainOptions,
executionProps: ExecutionProps // maybe with reference to avoid copy
}
```
The same idea applies to `PhysicalPlanner`, we usually just need `PhysicalOptimizeRules` or other information about plan.
https://github.com/apache/datafusion/blob/4bed04e4e312a0b125306944aee94a93c2ff6c4f/datafusion/core/src/physical_planner.rs#L364-L385
And, `create_initial_plan` in `DefaultPhysicalPlanner`, `ExecutionOptions ` is all we need, nothing else.
https://github.com/apache/datafusion/blob/4bed04e4e312a0b125306944aee94a93c2ff6c4f/datafusion/core/src/physical_planner.rs#L582-L585
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by reading the scan implementation in datafusion/core/src/datasource/memory.rs and the referenced functions in datafusion/core/src/physical_planner.rs: PhysicalPlanner, create_initial_plan, and DefaultPhysicalPlanner. Map which parts of SessionState each entry point actually uses and review the related dependency concerns in issue #11182. Done means the unnecessary SessionState dependencies are replaced with narrower interfaces without preserving the circular dependency.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100