[EPIC] Improve query planning speed
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
DataFusion can theoretically plan queries of almost any complexity. However, for moderately complex queries, especially those involving `UNION`S or deeply nested plans, it can sometimes take *seconds* to plan they query
At InfluxData we have had to disable certain equivalence checks because without doing so the planning time becomes exponential (and effectively locks up one of our cores)
Basically I would like to use this ticket to coordinate / start making meaningful progress reducing the planning time for DataFusion.
### Describe the solution you'd like
We have fairly thorough benchmarks at this point that I think covers many use cases.
```shell
# don't forget --profile=profiling otherwise you'll end up with no symbol names
cargo bench --profile=profiling --bench sql_planner
```
If anyone wants to help with this EPIC an excellent first step to do would be
1. Contribute example queries that are slow to plan in your environment
2. Spend time profiling the existing benchmarks to identify potential performance improvements
### Describe alternatives you've considered
Issues reporting slow planning time
- [x] https://github.com/apache/datafusion/issues/7698
- [ ] https://github.com/apache/datafusion/issues/17261
- [ ] https://github.com/apache/datafusion/issues/20002
Ideas to potentially increase performance
- [x] https://github.com/apache/datafusion/issues/9144
- [ ] https://github.com/apache/datafusion/issues/9577
- [x] https://github.com/apache/datafusion/issues/19796
- [x] https://github.com/apache/datafusion/issues/20495
Prior Epics
- [x] https://github.com/apache/datafusion/issues/13015
- [x] https://github.com/apache/datafusion/issues/5637
### Additional context
_No response_
Contributor guide
Research direction
Start by running `cargo bench --profile=profiling --bench sql_planner` and profiling the existing query-planning benchmarks with symbol names enabled. Compare the results with moderately complex or deeply nested queries from your environment, then use the identified bottleneck or slow example to define a focused performance change and benchmark improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100