apache / apache/datafusion

Improve join performance for h2o queries

Open
#13,765 0 comments 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

DataFusion joins are generally performant, but they can start erroring out when memory becomes limited.

Here are the h2o queries run on my local machine (Macbook M3 with 16 GB of RAM):

![h2o-join](https://github.com/user-attachments/assets/47046832-3e7a-4bd9-b024-b471c89798d8)

DataFusion performs really well except for query 5, which joins two 100 million row tables. DataFusion errors out for query 5 on my machine.

DataFusion is the fastest option when joining a 100 million row table with a 100 row or 100,000 row table.

These same queries are more performant in the official benchmarks which are run on a really powerful machine:

Screenshot 2024-12-13 at 1 36 28 PM

The official benchmarks show an error for DataFusion on the 1 billion row table:

Screenshot 2024-12-13 at 1 37 36 PM

So, I am not sure about the underlying issue, but seems like there are problems when memory becomes limited.

### Describe the solution you'd like

Hopefully DataFusion can perform similar to other engines for large table to large table joins.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing h2o query 5, which joins two 100-million-row tables, on the reported 16 GB MacBook setup and compare it with the official benchmark results. No source files, tests, or entry points are identified; done means large-table joins no longer fail under constrained memory and perform comparably to other engines.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.