apache / apache/datafusion

try to add local limit after the optimizer rule `join_selection`

Open
#10,170 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

In our cases, we have left_table(multi file group) and right_table(single file group), and we write the sql

```
select * from left_table join right_table on left_table.key = right_table.key
```

The final physical plan is not the optimal.

The init phy plan is below with **auto** join mode. The children of the join will be swapped by the rule `join_selection`
```
GlobalLimitExec: skip=0, fetch=10
ProjectionExec: ....
HashJoinExec: mode=**auto**, join_type=Inner, on=....
ProjectionExec: left_table
ParquetExec: file_groups={37 group: ....}
ProjectionExec: right_table
ParquetExec: file_groups={1 groups: ....}

```
after the physical optimizer, the most import rule is `join_selection`
The final plan is like:

```
....
GlobalLimitExec: skip=0, fetch=10
ProjectionExec: ....
HashJoinExec: **mode=CollectLeft,** join_type=Inner, on=....
ProjectionExec: right_table
ParquetExec: file_groups={1 group: ....}
ProjectionExec: left_table
ParquetExec: file_groups={37 groups: ....}
```

The optimal plan should be added a `LocalLimitExec` behind `GlobalLimitExec`

### Describe the solution you'd like

The optimal plan should be like

```
....
GlobalLimitExec: skip=0, fetch=10
LocalLimitExec:....
ProjectionExec: ....
HashJoinExec: **mode=CollectLeft,** join_type=Inner, on=....
ProjectionExec: right_table
ParquetExec: file_groups={1 group: ....}
ProjectionExec: left_table
ParquetExec: file_groups={37 groups: ....}
```

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing the physical optimizer rule `join_selection` and the `GlobalLimitExec` and `LocalLimitExec` plan nodes shown in the issue. Reproduce the SQL plan with a multi-file left table and single-file right table, then verify that the optimized plan places a local limit below the global limit while retaining the selected join mode.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.