datafusion-contrib / datafusion-contrib/datafusion-distributed
Investigate if ScalarSubqueryExec can replace network broadcasts
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 139
- Forks
- 67
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 35
Description
A ScalarSubqueryExec is functionally similar to a broadcast. In https://github.com/datafusion-contrib/datafusion-distributed/issues/180, we will likely add machinery to route updates from dynamic filters from produces to consumers. We can likely use this same machinery to make scalary subquery updates propagate across the network and potentially propagate broadcasts as well.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ScalarSubqueryExec in datafusion/physical-plan/src/scalar_subquery.rs and review the proposed dynamic-filter machinery in issue #180. Determine whether that mechanism can propagate scalar-subquery updates across the network and whether it can also support broadcasts. Done means documenting a concrete feasibility conclusion and the required scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100