apache / apache/arrow

[C++] Support hash-join on larger than memory datasets

Open
#31,769 5 comments 0 reactions 0 assignees View on GitHub
Component: C++ Status: needs champion Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

The current implementation of the hash-join node current queues in memory the hashtable, the entire build side input, and the entire probe side input (e.g. the entire dataset). This means the current implementation will run out of memory and crash if the input dataset is larger than the memory on the system.

By spilling to disk when memory starts to fill up we can allow the hash-join node to process datasets larger than the available memory on the machine.

**Reporter**: [Weston Pace](https://issues.apache.org/jira/browse/ARROW-16389) / @westonpace
#### Related issues:
- [[C++] Naive spillover implementation for join](https://github.com/apache/arrow/issues/29750) (supercedes)
#### PRs and other links:
- [GitHub Pull Request #13669](https://github.com/apache/arrow/pull/13669)

**Note**: *This issue was originally created as [ARROW-16389](https://issues.apache.org/jira/browse/ARROW-16389). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the superseding issue #29750 and the linked pull request #13669, then inspect the hash-join node described here. Done means hash joins can spill their in-memory state to disk and process datasets larger than available memory without crashing.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.