Optimize join ordering
- Dominant language
- Python
- Stars
- 89
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
**Problem**
Currently, we execute joins in the order they were given by the user. If the user does not pay attention, this can cause a significant performance penalty due to an unnecessary explosion of intermediate results.
**Solution**
We should automatically optimize the join ordering. Ideally, we have cardinality estimates for this from Parquet files or a metadata store, but we should also try to optimize join ordering without meaningful statistics. Possible approaches here include optimization based on partition counts or equivalence sets (https://blobs.duckdb.org/papers/tom-ebergen-msc-thesis-join-order-optimization-with-almost-no-statistics.pdf).
**Previous work**
We have already experimented with this, but never gotten to a working solution that we could merge (e.g., #809).
Contributor guide
Research direction
Start by reviewing the previous experiment in #809 and the linked DuckDB thesis. Investigate how joins are currently executed and how partition counts, equivalence sets, or metadata and cardinality estimates could guide ordering. Done means a working, mergeable automatic join-ordering solution, including cases without meaningful statistics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100