cube-js / cube-js/cube

BigQuery: Infinite loop of pre-aggregation warm-up when CPU quota is exceeded

Open
#2,506 0 comments 0 reactions 1 assignee Claimed by @ovr View on GitHub
driver:bigquery
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 10h
Merged PRs (30d)
203

Description

**Describe the bug**
When pre-aggregation warm-up is performed, it might fail because BigQuery won't provide enough CPU quota to the query. In this case, Cube.js will instantly try to restart the warm-up, and it has even fewer chances to succeed because the CPU quota will be even less. Thus, Cube.js pre-aggregation warm-up will be stuck in an infinite loop.

```
Error: Query exceeded resource limits. 150542 CPU seconds were used, and this query must use less than 98300 CPU seconds.
at BigQueryDriver.awaitForJobStatus (/cube/node_modules/@cubejs-backend/bigquery-driver/driver/BigQueryDriver.js:202:15)
at runMicrotasks ()
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at BigQueryDriver.waitForJobResult (/cube/node_modules/@cubejs-backend/bigquery-driver/driver/BigQueryDriver.js:225:22)
at PreAggregationLoader.refreshImplTempTableExternalStrategy (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/PreAggregations.ts:643:5)
at Object.query (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryCache.ts:269:26)
at QueryQueue.processQuery (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:249:21)
```

**To Reproduce**
Steps to reproduce the behavior:
1. Create heavy pre-aggregation.
2. Start Cube.js to initiate the warm-up.
3. Watch Cube.js failing the warm-up and restarting it in an infinite loop.

**Expected behavior**
Cube.js eventually creates the pre-aggregation or provides an actionable error message. Maybe exponential backoff might be used to help use the quota optimally.

**Version:**
Latest release channel in Cube Cloud as of today.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.