cube-js / cube-js/cube

"Please consider reducing total number of partitions by using rollup_lambda pre-aggregations" question

Open
#7,661 0 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
181

Description

**Problem**

We are seeing this warning in our refresh worker logs. After reading the documentation, it's not clear to me exactly how using a rollup_lambda will help the situation. Can you please explain a little more:

* How can one could leverage rollup_lambda to reduce the total number of partitions?
* Why would that help?
* Why is it recommending reducing the number of partitions when the largesst number of partitions in our pre-aggs is 24 (monthly, 2 years of data).

We are multi-tenant though, with 20 tenants, if that changes things. But even if it is 24 * 20, that's still at the low end of the upper bound range recommended in the docs (500-1000).

We are running a production workload with:
* 2x API
* 1x Refresh Worker
* 1x Cubestore Router
* 4x Cubestore Workers

We are running on Azure AKS, with Cubestore data residing in Azure Blobstore LRS Premium, provided by Azure Blobfuse Premium storage class persistent volume. Container cpu & mem sizing is as per recommendations in the docs.

Thank you!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.