apache / apache/datafusion

Consolidate optimizer passes to improve planning speed

Open
#15,045 3 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

@jayzhan211 removed the `UnwrapCastInComparison` optimizer (and combined it with the simplifier) In
- https://github.com/apache/datafusion/pull/15012

Doing so seemed to make a non trivial difference for planning speed:
- https://github.com/apache/datafusion/pull/15012#issuecomment-2701077462

I *think* the improvement came from reducing the number of Optimizer passes (and this rewrites/copies of the plan and all expressions) that happened

Here were my mesurements about speed
Image

### Describe the solution you'd like

I would like to make planning faster by potentially combining other passes from this list:

https://github.com/apache/datafusion/blob/43ecd9b807877946706628633308f73a4645de1f/datafusion/optimizer/src/optimizer.rs#L243-L272

### Describe alternatives you've considered

Some potential candidates to try consolidating:
* `EliminateNestedUnion` + `EliminateOneUnion`
* `EliminateJoin` and `EliminateJoin`

You can run the planning benchmarks like
```shell
cargo bench --bench sql_planner
```

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the optimizer pass list in datafusion/optimizer/src/optimizer.rs and review the prior UnwrapCastInComparison consolidation in PR 15012. Run cargo bench --bench sql_planner to establish planning-speed measurements, then evaluate candidate pass combinations such as the union passes and compare the resulting benchmark behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.