apache / apache/datafusion

Add CollectLeft partition mode to SortMergeJoinExec

Open
#23,112 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

SortMergeJoinExec currently always runs in a symmetric, hash-partitioned mode:
both inputs are hash-partitioned on the join keys and sorted, and partition i
of the left is merge-joined with partition i of the right.

When one side is small, hash-repartitioning the large side is wasteful. For
hash joins this is already handled by PartitionMode::CollectLeft (collect the
small build side once, broadcast it, don't repartition the probe side). This PR
brings the same idea to sort-merge joins.

### Describe the solution you'd like

Implement CollectLeft mode for SMJ

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing PartitionMode::CollectLeft implementation for hash joins and the SortMergeJoinExec execution path. Verify how the small left side is collected and broadcast, then compare sort-merge join behavior with symmetric partitioning; done means CollectLeft avoids repartitioning the large side while producing correct join results.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.