BigQuery: Infinite loop of pre-aggregation warm-up when CPU quota is exceeded
- Dominant language
- Rust
- Stars
- 20.8k
- Forks
- 2.1k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 203
Description
**Describe the bug**
When pre-aggregation warm-up is performed, it might fail because BigQuery won't provide enough CPU quota to the query. In this case, Cube.js will instantly try to restart the warm-up, and it has even fewer chances to succeed because the CPU quota will be even less. Thus, Cube.js pre-aggregation warm-up will be stuck in an infinite loop.
```
Error: Query exceeded resource limits. 150542 CPU seconds were used, and this query must use less than 98300 CPU seconds.
at BigQueryDriver.awaitForJobStatus (/cube/node_modules/@cubejs-backend/bigquery-driver/driver/BigQueryDriver.js:202:15)
at runMicrotasks ()
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at BigQueryDriver.waitForJobResult (/cube/node_modules/@cubejs-backend/bigquery-driver/driver/BigQueryDriver.js:225:22)
at PreAggregationLoader.refreshImplTempTableExternalStrategy (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/PreAggregations.ts:643:5)
at Object.query (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryCache.ts:269:26)
at QueryQueue.processQuery (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:249:21)
```
**To Reproduce**
Steps to reproduce the behavior:
1. Create heavy pre-aggregation.
2. Start Cube.js to initiate the warm-up.
3. Watch Cube.js failing the warm-up and restarting it in an infinite loop.
**Expected behavior**
Cube.js eventually creates the pre-aggregation or provides an actionable error message. Maybe exponential backoff might be used to help use the quota optimally.
**Version:**
Latest release channel in Cube Cloud as of today.
Contributor guide
Assessment
This issue has not been assessed yet.