apache / apache/pinot

[discuss]integrating Apache DataSketches library

Open
#6,420 4 comments 9 reactions 0 assignees View on GitHub
feature
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

The following is the discussion with Mayank on slack:

Mark: Hi Team, I have seen that in 0.4.0, pinot has implemented the initial version of theta-sketch based distinct count aggregation function, utilizing the Apache DataSketches library. Compared to Druid the latest release which has also included DataSketches extension(Theta sketch, Tuple sketch, Quantiles sketch ,HLL sketch), pinot has any plan to implement other sketchs other than Theta sketch). Thanks.

Mayank: Pinot already supports HLL and TDigest based percentiles. If there's a specific case where you would find DataSketch based implementations more useful, we can definitely explore that. If so, would recommend filing an issue for that.

Mayank: For HLL we use com.clearspring.analytics.stream.cardinality.HyperLogLog,And for TDigest, we use com.tdunning.math.stats.TDigest

Mark: we maybe need to pay attention to KLL sketch vs t-digest(pinot impmentation) and seeing the following comparison by datasketches, https://datasketches.apache.org/docs/Quantiles/KllSketchVsTDigest.html

Mayank: Thanks for sharing @Mark.Tang. We can definitely explore adding these if needed.

Mark: appendix(https://github.com/apache/datasketches-website/blob/master/docs/pdf/DataSketches_deck.pdf): HLL
![pinot1](https://user-images.githubusercontent.com/187414/103863413-c65e1500-50fb-11eb-9c6a-b1b9677b69a7.png)

Also noting that DataSketches includes a latest CPC Sketch: Estimating Stream Cardinalities more efficiently than the famous HLL sketch, which is from https://arxiv.org/pdf/1708.06839.pdf

Contributor guide

Open the contributing guide

Research direction

The discussion mentions Pinot's existing HLL and TDigest implementations but names no source files or tests. Start by reviewing those aggregation implementations and comparing the proposed DataSketches options, especially KLL and CPC. Done requires choosing a specific sketch and use case, then defining a concrete implementation scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.