cube-js / cube-js/cube

KsqlDriver pre-aggregations return 400 error while loading

Open
#6,746 0 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
181

Description

**Describe the bug**
When using a Cube with a KSQL datasource and a pre-aggregation, there appears to be some sort of race condition when querying. I know my Cube configuration is correct because I can eventually successfully query from this KSQL cube. The problem is with the first time the KSQL cube is queried, and the pre-aggregation is not yet loaded. In this case, a 400 error is returned from the query request. Eventually, a 200 will be returned once the pre-aggregation is established.

This differs from other pre-aggregations such as ones built using a MSSQL driver. In those pre-aggregations, a query which matches a pre-aggregation will wait until the matching pre-aggregation is built before returning data and a status of 200. Or I think you get a 200 with a "Continue Wait...". Something like that, but NOT a 400 error.

**Expected behavior**
When a query matches a pre-aggregation for a cube, the data from the pre-aggregation should be returned with an http status of 200. When the pre-aggregation is not loaded yet, the query should wait for the pre-agg to be loaded, or return 200 with no data? Not sure about that part, but I'm pretty sure we don't want to a 400 error to be returned.

**Version:**
0.33.25

**Additional context**
Here are the logs from the cube-api instance when the error occurs:
{
"message": "Load PreAggregations Tables",
"preAggregationId": "KPolicy.policyDetailsRollup",
"timezone": "UTC",
"timestampFormat": "YYYY-MM-DD[T]HH:mm:ss.SSS[Z]",
"tableName": "prod_pre_aggregations.k_policy_policy_details_rollup",
"invalidateKeyQueries": [
[
"SELECT FLOOR((UNIX_TIMESTAMP()) / 3600) as refresh_key",
[],
{
"external": true,
"renewalThreshold": 10
}
]
],
"partitionInvalidateKeyQueries": [],
"type": "rollup",
"external": true,
"previewSql": [
"SELECT * FROM prod_pre_aggregations.k_policy_policy_details_rollup LIMIT 1000",
[]
],
"preAggregationsSchema": "prod_pre_aggregations",
"loadSql": [
"CREATE TABLE `prod_pre_aggregations.k_policy_policy_details_rollup` WITH (KEY_FORMAT='JSON') AS SELECT\n `k_policy`.dbpEntityId `k_policy__dbp_entity_id`, `k_policy`.dbpId `k_policy__dbp_id`, `k_policy`.description `k_policy__description`, `k_policy`.id `k_policy__id`, `k_policy`.latest `k_policy__latest`, `k_policy`.name `k_policy__name`, count(`k_policy`.policyid) `k_policy__count`\n FROM\n policy_table AS `k_policy` GROUP BY `k_policy`.dbpEntityId, `k_policy`.dbpId, `k_policy`.description, `k_policy`.id, `k_policy`.latest, `k_policy`.name",
[]
],
"sql": [
"SELECT\n `k_policy`.dbpEntityId `k_policy__dbp_entity_id`, `k_policy`.dbpId `k_policy__dbp_id`, `k_policy`.description `k_policy__description`, `k_policy`.id `k_policy__id`, `k_policy`.latest `k_policy__latest`, `k_policy`.name `k_policy__name`, count(`k_policy`.policyid) `k_policy__count`\n FROM\n policy_table AS `k_policy` GROUP BY `k_policy`.dbpEntityId, `k_policy`.dbpId, `k_policy`.description, `k_policy`.id, `k_policy`.latest, `k_policy`.name",
[]
],
"uniqueKeyColumns": [
"`k_policy__dbp_entity_id`",
"`k_policy__dbp_id`",
"`k_policy__description`",
"`k_policy__id`",
"`k_policy__latest`",
"`k_policy__name`"
],
"aggregationsColumns": [
"sum(`k_policy__count`)"
],
"dataSource": "ksqldb",
"indexesSql": [],
"createTableIndexes": [],
"readOnly": false,
"streamOffset": "earliest"
},
"requestId": "26fc65a8-86ff-4937-99cc-9effefa6243b-span-1",
"duration": 7
}

{
"message": "Error querying db",
"query": "SELECT `k_policy__dbp_entity_id` `k_policy__dbp_entity_id`, `k_policy__dbp_id` `k_policy__dbp_id`, `k_policy__name` `k_policy__name`, `k_policy__latest` `k_policy__latest` FROM prod_pre_aggregations.k_policy_policy_details_rollup AS `k_policy__policy_details_rollup` WHERE (`k_policy__dbp_entity_id` IN (?, ?)) GROUP BY 1, 2, 3, 4 ORDER BY 3 ASC LIMIT 10000",
"params": [
"1",
"2"
],
"error": "Error: No pre-aggregation partitions were built yet for the pre-aggregation serving this query and this API instance wasn't set up to build pre-aggregations. Please make sure your refresh worker is configured correctly, running, pre-aggregation tables are built and all pre-aggregation refresh settings like timezone match. Expected table name patterns: prod_pre_aggregations.k_policy_policy_details_rollup_*_fitaehcb_*
at PreAggregationLoader.loadPreAggregation (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/PreAggregations.ts:601:17)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at preAggregationPromise (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/PreAggregations.ts:2168:30)
at QueryOrchestrator.fetchQuery (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryOrchestrator.ts:241:9)
at OrchestratorApi.executeQuery (/cube/node_modules/@cubejs-backend/server-core/src/core/OrchestratorApi.ts:98:20)
at /cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1351:21
at async Promise.all (index 0)
at ApiGateway.getSqlResponseInternal (/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1349:31)
at /cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1528:28
at async Promise.all (index 0)
at ApiGateway.load (/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1519:23)
at /cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:245:7",
"requestId": "26fc65a8-86ff-4937-99cc-9effefa6243b-span-1"
}

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.