cube-js / cube-js/cube

Quantile Support for Pre-Aggregations

Open
#2,931 5 comments 4 reactions 2 assignees Claimed by @kmgreen2 View on GitHub
help wanted pre-aggregations
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
181

Description

**Is your feature request related to a problem? Please describe.**
Much of our aggregations are based on quantiles (e.g. 0.95 -> 95p). It is well-known that you cannot merge quantiles without having all of the original data, which prevents such aggregations in pre-aggregations. It would be very nice to have pre-computed quantiles that can be merged, similar to counts, sums, etc.

First, does the community care about quantile-based pre-aggregations? If the answer is no, then we can just close this out.

If there seems to be a use for this, I'd be happy to help.

I mentioned this at the community meeting on 06/09/2021 (@rpaik).

**Describe the solution you'd like**

References:

- T-digest paper: https://arxiv.org/pdf/1902.04023.pdf
- T-digest reference implementation (there is a JS port): https://github.com/tdunning/t-digest

There are digest-based data structures, such as t-digests, that can produce accurate, bounded-size, *mergeable* quantile approximations. They are typically used in streaming or map-reduce-like settings for efficiently/concurrently combining rank statistics, but I think they would also be useful in pre-aggregations. There is a definite tradeoff between space and accuracy, so tuning could be tricky, but it seems that there are decent defaults that would suit most use cases.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.