"Please consider reducing total number of partitions by using rollup_lambda pre-aggregations" question
- Dominant language
- Rust
- Stars
- 20.8k
- Forks
- 2.1k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 181
Description
**Problem**
We are seeing this warning in our refresh worker logs. After reading the documentation, it's not clear to me exactly how using a rollup_lambda will help the situation. Can you please explain a little more:
* How can one could leverage rollup_lambda to reduce the total number of partitions?
* Why would that help?
* Why is it recommending reducing the number of partitions when the largesst number of partitions in our pre-aggs is 24 (monthly, 2 years of data).
We are multi-tenant though, with 20 tenants, if that changes things. But even if it is 24 * 20, that's still at the low end of the upper bound range recommended in the docs (500-1000).
We are running a production workload with:
* 2x API
* 1x Refresh Worker
* 1x Cubestore Router
* 4x Cubestore Workers
We are running on Azure AKS, with Cubestore data residing in Azure Blobstore LRS Premium, provided by Azure Blobfuse Premium storage class persistent volume. Container cpu & mem sizing is as per recommendations in the docs.
Thank you!
Contributor guide
Assessment
This issue has not been assessed yet.