Consolidate optimizer passes to improve planning speed
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
@jayzhan211 removed the `UnwrapCastInComparison` optimizer (and combined it with the simplifier) In
- https://github.com/apache/datafusion/pull/15012
Doing so seemed to make a non trivial difference for planning speed:
- https://github.com/apache/datafusion/pull/15012#issuecomment-2701077462
I *think* the improvement came from reducing the number of Optimizer passes (and this rewrites/copies of the plan and all expressions) that happened
Here were my mesurements about speed
### Describe the solution you'd like
I would like to make planning faster by potentially combining other passes from this list:
https://github.com/apache/datafusion/blob/43ecd9b807877946706628633308f73a4645de1f/datafusion/optimizer/src/optimizer.rs#L243-L272
### Describe alternatives you've considered
Some potential candidates to try consolidating:
* `EliminateNestedUnion` + `EliminateOneUnion`
* `EliminateJoin` and `EliminateJoin`
You can run the planning benchmarks like
```shell
cargo bench --bench sql_planner
```
### Additional context
_No response_
Contributor guide
Research direction
Start with the optimizer pass list in datafusion/optimizer/src/optimizer.rs and review the prior UnwrapCastInComparison consolidation in PR 15012. Run cargo bench --bench sql_planner to establish planning-speed measurements, then evaluate candidate pass combinations such as the union passes and compare the resulting benchmark behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100