apache / apache/datafusion

Optimize the join operators

Open
#16,710 32 comments 5 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

DataFusion is underperforming the Polars streaming engine on some localhost join queries (1e8 rows of data on a Macbook M3 with 16GB of RAM):

Image

Here are the [join queries](https://github.com/apache/datafusion/blob/main/benchmarks/queries/h2o/join.sql).

I am guessing the join operator can be optimized, similar to how the filtering and aggregation operations were optimized.

Here is an example of how the median function was made faster: https://github.com/apache/datafusion/issues/13550

See this epic for more info: https://github.com/apache/datafusion/issues/13548

Contributor guide

Open the contributing guide

Research direction

Start with the H2O join queries in benchmarks/queries/h2o/join.sql and the linked epic for context on the performance work. The issue does not identify a specific join operator or optimization, so first investigate which queries lag and compare with the approach in issue 13550. Done would mean measurable improvement on the reported join workload; no target is specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.