apache / apache/datafusion-ballista
Implement broadcast join optimization
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
When we support broadcast exchanges https://github.com/apache/arrow-ballista/issues/342 we can transform certain joins to utilize it.
**Describe the solution you'd like**
Currently all plans involving hash joins look like the following.
```
HashJoin <- RemoteExchange (partitioned) <- build side input
<- RemoteExchange (partitioned) <- probe side input
```
When the build side is small (e.g. Spark uses 10MB * number of partitions for this by default - but generally bigger can help as well in my experience).
The new plan after optimization looks like this (note the missing exchange in the probe side, that side doesn't require shuffling now)
```
HashJoin <- BroadcastExchange <- build side input
<- probe side input
```
**Describe alternatives you've considered**
Implement the (physical) optimization rule. The rule should run after the `HashBuildProbeOrder` rule from DataFusion.
**Additional context**
Contributor guide
Research direction
Start by reading issue #342 for the broadcast-exchange behavior, then inspect DataFusion's HashBuildProbeOrder rule and the physical optimization-rule registration point. Done means a rule selects broadcast joins for sufficiently small build sides, runs after HashBuildProbeOrder, and produces the plan shape described in this issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100