[discuss]integrating Apache DataSketches library
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
The following is the discussion with Mayank on slack:
Mark: Hi Team, I have seen that in 0.4.0, pinot has implemented the initial version of theta-sketch based distinct count aggregation function, utilizing the Apache DataSketches library. Compared to Druid the latest release which has also included DataSketches extension(Theta sketch, Tuple sketch, Quantiles sketch ,HLL sketch), pinot has any plan to implement other sketchs other than Theta sketch). Thanks.
Mayank: Pinot already supports HLL and TDigest based percentiles. If there's a specific case where you would find DataSketch based implementations more useful, we can definitely explore that. If so, would recommend filing an issue for that.
Mayank: For HLL we use com.clearspring.analytics.stream.cardinality.HyperLogLog,And for TDigest, we use com.tdunning.math.stats.TDigest
Mark: we maybe need to pay attention to KLL sketch vs t-digest(pinot impmentation) and seeing the following comparison by datasketches, https://datasketches.apache.org/docs/Quantiles/KllSketchVsTDigest.html
Mayank: Thanks for sharing @Mark.Tang. We can definitely explore adding these if needed.
Mark: appendix(https://github.com/apache/datasketches-website/blob/master/docs/pdf/DataSketches_deck.pdf): HLL

Also noting that DataSketches includes a latest CPC Sketch: Estimating Stream Cardinalities more efficiently than the famous HLL sketch, which is from https://arxiv.org/pdf/1708.06839.pdf
Contributor guide
Research direction
The discussion mentions Pinot's existing HLL and TDigest implementations but names no source files or tests. Start by reviewing those aggregation implementations and comparing the proposed DataSketches options, especially KLL and CPC. Done requires choosing a specific sketch and use case, then defining a concrete implementation scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100