apache / apache/datafusion

Remove datafusion.optimizer.physical_uncorrelated_scalar_subquery config option

Open
#22,566 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

We had to add the session `property datafusion.optimizer.physical_uncorrelated_scalar_subquery` in https://github.com/apache/datafusion/pull/22530 because the PR introduces a node to execute subqueries, `ScalarSubqueryExec`, whose structure holds shared state.

More context can be found in the following comment: https://github.com/datafusion-contrib/datafusion-distributed/issues/460#issuecomment-4534585708

> If ScalarSubqueryExec and the node holding the ScalarSubqueryExpr have a network boundary between them (i.e. they are on different machines), the query fails during deserialization with the following error: Internal("ScalarSubqueryExpr can only be deserialized as part of a surrounding ScalarSubqueryExec"). Even if we edited the decoder to create a fresh ScalarSubqueryResults on the worker, I don’t think the result would be correct: the worker’s slots are backed by a different Arc from the coordinator’s, so the writer would never fill them.

This affects both datafusion-distributed and Ballista (and we suspect in any distributed query engine that ends up in the situation mentioned above). Because this is a breaking change for them, we decided to gate planning of the new node (instead of the previous LeftJoin) behind a config option.

Ideally, we want to maintain this flag for a few more releases and then deprecate it (removing support for LeftJoin execution for scalar subqueries)

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing the session property introduced in PR 22530 and the planning paths for ScalarSubqueryExec and the previous LeftJoin implementation. Review the distributed-query compatibility context linked in the issue, then identify the release and deprecation constraints; done means the option and its legacy planning support are consistently handled without breaking affected engines.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, distributed-systems
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.