apache / apache/datafusion

Consider overlapping scalar subquery and parent query computation

Open
#21,591 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Right now (if/when #21240 lands), uncorrelated scalar subqueries are evaluated eagerly and in parallel, and then the parent query is evaluated. We could consider some alternative strategies, such as:

1. Continue evaluating scalar subqueries eagerly and in parallel, but also allow parent query computation to be overlapped with subquery computation. This would require a synchronization point between ScalarSubqueryExpr evaluation and the evaluation of the corresponding subquery itself (via ScalarSubqueryExec); if the former happens before the latter, we'll need to arrange to block parent query evaluation until the subquery results are available.
2. Alternatively, we could explore lazily evaluating scalar subqueries. This might exploit parallelism less effectively, but it would avoid wasting work (e.g., in the case that a subquery doesn't ever need to be evaluated, e.g., due to another predicate in the `WHERE` clause).

Not sure how either of these approaches interacts with morsel-driven parallelism...

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with ScalarSubqueryExpr and ScalarSubqueryExec, then trace how uncorrelated scalar subqueries and parent queries are evaluated in parallel. Examine how morsel-driven parallelism affects the two proposed strategies. Done requires selecting and implementing a strategy, including the needed synchronization or lazy evaluation behavior, but the issue does not define which alternative to choose.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.