Optimize the join operators
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
DataFusion is underperforming the Polars streaming engine on some localhost join queries (1e8 rows of data on a Macbook M3 with 16GB of RAM):
Here are the [join queries](https://github.com/apache/datafusion/blob/main/benchmarks/queries/h2o/join.sql).
I am guessing the join operator can be optimized, similar to how the filtering and aggregation operations were optimized.
Here is an example of how the median function was made faster: https://github.com/apache/datafusion/issues/13550
See this epic for more info: https://github.com/apache/datafusion/issues/13548
Contributor guide
Research direction
Start with the H2O join queries in benchmarks/queries/h2o/join.sql and the linked epic for context on the performance work. The issue does not identify a specific join operator or optimization, so first investigate which queries lag and compare with the approach in issue 13550. Done would mean measurable improvement on the reported join workload; no target is specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100