apache / apache/datafusion

Specialized `GroupValues` for `primitive` and `large_primitive`

Open
#16,135 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

After #15961, in `GroupValuesPrimitive`, we store the `hash` of group to speed up rehashing.

However, I found the bottleneck of `rehashing` is not hash computation, it is actually randomly access to the `values`.

So, after experimenting, the better is:
- We directly store the `group value` in `map`
- So, during probing we can perform `eq` in-place
- And, during rehashing we just perform the cheap hash computation for `group value`

But for the `large primitives`, it may be still better to store their hashes in `map` rather than themselves, otherwise it will make `map` too large and lead to possible regression.

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing GroupValuesPrimitive and its map access during probing and rehashing. Compare storing group values with storing hashes for large primitives, then validate that the specialized primitive and large_primitive behavior improves rehashing without making the map too large or causing a regression.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.