apache / apache/datafusion

Consolidate MergeJoin with HashJoin to adaptive join relations according to runtime resources and table sizes

Open
#2,316 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

A possible solution I could think of currently:
1. Always choose to use HashJoin when there is no statistical information indicating that both tables are large.
2. Memory tracking while building hashtable for building side.
3. When the hash-builder fails to grow its memory
3.1. sort and spill the in-memory hashtable into spill0, free memory.
3.2. buffer and sort the incoming records for the buffer table until it's exhausted, do a sort.
3.3. buffer and sort the records for the streaming side until it's finished, do a sort.
3.4 MergeJoin the two sides.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing HashJoin and MergeJoin implementations and the hash-builder memory-growth path. Trace how spill0, the buffer table, and the streaming side could be sorted, then define completion around choosing or falling back to MergeJoin based on runtime memory and table-size information.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.