Add HLL++ support for better accuracy and possibly lower memory cost
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
DISTINCTCOUNTHLL accuracy and memory footprint can be improved through latest HLL algorithms. We have a choice either replace the existing implementation with a better one or leave the existing DISTINCTCOUNTHLL to implement original HLL and create separate functions (ex: DISTINCTCOUNTHLLPLUSPLUS).
[Google's HLL++](http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/40671.pdf) -- a popular algorithm amongst the community offers lot of improvements over original HLL. There are multiple java implementations of HLL++. Most of them have variations in performance due to the register size and other implementation choices. Clearspring [stream-lib](https://github.com/addthis/stream-lib) used for current HyperLogLog function, implements HLL++ as [HyperLogPlus](https://github.com/addthis/stream-lib/commits/master/src/main/java/com/clearspring/analytics/stream/cardinality/HyperLogLogPlus.java).
Contributor guide
Research direction
Start with Pinot’s existing HyperLogLog function and compare it with Clearspring stream-lib’s HyperLogPlus implementation and the linked HLL++ paper. Resolve whether to replace DISTINCTCOUNTHLL or add a separate function, then verify the chosen approach improves accuracy and memory footprint.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100