apache / apache/druid

Replace Datasketch HLL with newer Datasketch CPC sketch

Open
#7,269 8 comments 0 reactions 0 assignees View on GitHub
Design Review Proposal
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Motivation

the datasketches library has a new Unique Counting Sketch called CPC sketch that has better accuracy per size than HLL.

https://github.com/DataSketches/sketches-core/releases

### Proposed changes

replace Datasketch HLL sketch with CPC or offer it alongside as a higher accuracy sketch

### Rationale

Better accuracy

### Operational impact

The docs of datasketches don't describe the whether CPC algorithm is more CPU intensive or not. This will determine whether we want to completely deprecate DataSketches HLL sketch and replace it with CPC or keep CPC as an additional option.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the linked DataSketches releases and the existing Druid integration for the HLL sketch; the issue does not name files or tests. Compare CPC and HLL accuracy and CPU impact, then determine whether CPC should replace HLL or be offered alongside it. Done means the chosen integration and its operational trade-offs are documented and covered by appropriate tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.