Quantile Support for Pre-Aggregations
- Dominant language
- Rust
- Stars
- 20.8k
- Forks
- 2.1k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 181
Description
**Is your feature request related to a problem? Please describe.**
Much of our aggregations are based on quantiles (e.g. 0.95 -> 95p). It is well-known that you cannot merge quantiles without having all of the original data, which prevents such aggregations in pre-aggregations. It would be very nice to have pre-computed quantiles that can be merged, similar to counts, sums, etc.
First, does the community care about quantile-based pre-aggregations? If the answer is no, then we can just close this out.
If there seems to be a use for this, I'd be happy to help.
I mentioned this at the community meeting on 06/09/2021 (@rpaik).
**Describe the solution you'd like**
References:
- T-digest paper: https://arxiv.org/pdf/1902.04023.pdf
- T-digest reference implementation (there is a JS port): https://github.com/tdunning/t-digest
There are digest-based data structures, such as t-digests, that can produce accurate, bounded-size, *mergeable* quantile approximations. They are typically used in streaming or map-reduce-like settings for efficiently/concurrently combining rank statistics, but I think they would also be useful in pre-aggregations. There is a definite tradeoff between space and accuracy, so tuning could be tricky, but it seems that there are decent defaults that would suit most use cases.
Contributor guide
Assessment
This issue has not been assessed yet.