apache / apache/datafusion

Use roaring bitmap in hash join for tracking hash collisions

Open
#4,093 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

to improve memory efficiency at a cost of compression, but possibly can allow a wider range of join schemes to be executed without backing to sort merge join

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by locating DataFusion's hash join implementation and its collision-tracking logic, then determine how a roaring bitmap would affect memory use and join execution; done requires an agreed design and validation that the relevant join schemes still work.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.