Remove datafusion.optimizer.physical_uncorrelated_scalar_subquery config option
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
We had to add the session `property datafusion.optimizer.physical_uncorrelated_scalar_subquery` in https://github.com/apache/datafusion/pull/22530 because the PR introduces a node to execute subqueries, `ScalarSubqueryExec`, whose structure holds shared state.
More context can be found in the following comment: https://github.com/datafusion-contrib/datafusion-distributed/issues/460#issuecomment-4534585708
> If ScalarSubqueryExec and the node holding the ScalarSubqueryExpr have a network boundary between them (i.e. they are on different machines), the query fails during deserialization with the following error: Internal("ScalarSubqueryExpr can only be deserialized as part of a surrounding ScalarSubqueryExec"). Even if we edited the decoder to create a fresh ScalarSubqueryResults on the worker, I don’t think the result would be correct: the worker’s slots are backed by a different Arc from the coordinator’s, so the writer would never fill them.
This affects both datafusion-distributed and Ballista (and we suspect in any distributed query engine that ends up in the situation mentioned above). Because this is a breaking change for them, we decided to gate planning of the new node (instead of the previous LeftJoin) behind a config option.
Ideally, we want to maintain this flag for a few more releases and then deprecate it (removing support for LeftJoin execution for scalar subqueries)
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by tracing the session property introduced in PR 22530 and the planning paths for ScalarSubqueryExec and the previous LeftJoin implementation. Review the distributed-query compatibility context linked in the issue, then identify the release and deprecation constraints; done means the option and its legacy planning support are consistently handled without breaking affected engines.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, distributed-systems
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100