roaringbitmap aggregation and deserialization
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
### Background
For use case like the dashboard, we have the requirements to precompute metrics for several years so we can enable the year_over_year comparison. One of the biggest challenges for the metric precomputation is precompute non-rollup metrics. Sometimes we can use the hyperloglog to get the estimated number, but here we want to discuss how to support precise non-rollup metric calculation. e.g. count(distinct) using the bitmap. (Apache Kylin also has similar [implementation](https://kylin.apache.org/blog/2016/08/01/count-distinct-in-kylin/#:~:text=Apache%20Kylin%20implements%20approximately%20count,max%20error%20rate%20is%201.22%25.))
### Problem statement
Currently the Pinot already provide the [bitmap way](https://docs.pinot.apache.org/configuration-reference/functions/distinctcountbitmap) for count(distinct), but it does not support:
1. stored the query result as the bitmap (binary data type) for further aggregation.
2. calculate the cardinality of bitmap (stored as binary data type).
3. bitmap aggregation: merge two bitmap into new a bitmap when we roll up the metric for finest granularity to higher granularity
Contributor guide
Assessment
This issue has not been assessed yet.