dask / dask/dask-expr

Optimize join ordering

Open
#1,065 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
89
Forks
26
PR merge metrics
No merged PRs in 30d

Description

**Problem**

Currently, we execute joins in the order they were given by the user. If the user does not pay attention, this can cause a significant performance penalty due to an unnecessary explosion of intermediate results.

**Solution**

We should automatically optimize the join ordering. Ideally, we have cardinality estimates for this from Parquet files or a metadata store, but we should also try to optimize join ordering without meaningful statistics. Possible approaches here include optimization based on partition counts or equivalence sets (https://blobs.duckdb.org/papers/tom-ebergen-msc-thesis-join-order-optimization-with-almost-no-statistics.pdf).

**Previous work**

We have already experimented with this, but never gotten to a working solution that we could merge (e.g., #809).

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the previous experiment in #809 and the linked DuckDB thesis. Investigate how joins are currently executed and how partition counts, equivalence sets, or metadata and cardinality estimates could guide ordering. Done means a working, mergeable automatic join-ordering solution, including cases without meaningful statistics.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.