Offheap execution for DISTINCT, DISTINCT_COUNT, DISTINCT_SUM and DISTINCT_AVG
Open
in-progress
performance
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
- Current distinct(count) functions create in-memory sets.
- Increased chances of OOM
- Incurs gc pressure
- Cannot handle high cardinality
- Hard to utilize disk for spilling
- Off-heap (direct buffer) hash table based solution can help here.
- Can be extended by supporting spilling over to disk
- Off heap hash-table can potentially be extended to group-by queries.
Contributor guide
Assessment
This issue has not been assessed yet.