Consolidate MergeJoin with HashJoin to adaptive join relations according to runtime resources and table sizes
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
A possible solution I could think of currently:
1. Always choose to use HashJoin when there is no statistical information indicating that both tables are large.
2. Memory tracking while building hashtable for building side.
3. When the hash-builder fails to grow its memory
3.1. sort and spill the in-memory hashtable into spill0, free memory.
3.2. buffer and sort the incoming records for the buffer table until it's exhausted, do a sort.
3.3. buffer and sort the records for the streaming side until it's finished, do a sort.
3.4 MergeJoin the two sides.
Contributor guide
Research direction
Start by reading the existing HashJoin and MergeJoin implementations and the hash-builder memory-growth path. Trace how spill0, the buffer table, and the streaming side could be sorted, then define completion around choosing or falling back to MergeJoin based on runtime memory and table-size information.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100