apache / apache/datafusion-ballista

Implement broadcast join optimization

Open
#348 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
When we support broadcast exchanges https://github.com/apache/arrow-ballista/issues/342 we can transform certain joins to utilize it.

**Describe the solution you'd like**

Currently all plans involving hash joins look like the following.

```
HashJoin <- RemoteExchange (partitioned) <- build side input
<- RemoteExchange (partitioned) <- probe side input
```

When the build side is small (e.g. Spark uses 10MB * number of partitions for this by default - but generally bigger can help as well in my experience).

The new plan after optimization looks like this (note the missing exchange in the probe side, that side doesn't require shuffling now)

```
HashJoin <- BroadcastExchange <- build side input
<- probe side input
```

**Describe alternatives you've considered**

Implement the (physical) optimization rule. The rule should run after the `HashBuildProbeOrder` rule from DataFusion.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by reading issue #342 for the broadcast-exchange behavior, then inspect DataFusion's HashBuildProbeOrder rule and the physical optimization-rule registration point. Done means a rule selects broadcast joins for sufficiently small build sides, runs after HashBuildProbeOrder, and produces the plan shape described in this issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.