apache / apache/datafusion

[EPIC] Improve query planning speed

Open
#19,795 2 comments 2 reactions 0 assignees View on GitHub
enhancement EPIC PROPOSAL EPIC
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

DataFusion can theoretically plan queries of almost any complexity. However, for moderately complex queries, especially those involving `UNION`S or deeply nested plans, it can sometimes take *seconds* to plan they query

At InfluxData we have had to disable certain equivalence checks because without doing so the planning time becomes exponential (and effectively locks up one of our cores)

Basically I would like to use this ticket to coordinate / start making meaningful progress reducing the planning time for DataFusion.

### Describe the solution you'd like

We have fairly thorough benchmarks at this point that I think covers many use cases.

```shell
# don't forget --profile=profiling otherwise you'll end up with no symbol names
cargo bench --profile=profiling --bench sql_planner
```

If anyone wants to help with this EPIC an excellent first step to do would be
1. Contribute example queries that are slow to plan in your environment
2. Spend time profiling the existing benchmarks to identify potential performance improvements

### Describe alternatives you've considered

Issues reporting slow planning time
- [x] https://github.com/apache/datafusion/issues/7698
- [ ] https://github.com/apache/datafusion/issues/17261
- [ ] https://github.com/apache/datafusion/issues/20002

Ideas to potentially increase performance
- [x] https://github.com/apache/datafusion/issues/9144
- [ ] https://github.com/apache/datafusion/issues/9577
- [x] https://github.com/apache/datafusion/issues/19796
- [x] https://github.com/apache/datafusion/issues/20495

Prior Epics
- [x] https://github.com/apache/datafusion/issues/13015
- [x] https://github.com/apache/datafusion/issues/5637

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by running `cargo bench --profile=profiling --bench sql_planner` and profiling the existing query-planning benchmarks with symbol names enabled. Compare the results with moderately complex or deeply nested queries from your environment, then use the identified bottleneck or slow example to define a focused performance change and benchmark improvement.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.