cube-js / cube-js/cube

Pre-Aggregations missing data

Open
#11,203 2 comments 0 reactions 0 assignees View on GitHub
cube store driver:oracle pre-aggregations
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
181

Description

I am running Cube Core with Oracle as a source database and Minio for pre-aggregation storage. I am running into an issue that causes pre-aggregations to not contain all data that they should. It seems ot be driven by the size of the partition.

Below is an example of a pre.aggregation. The below example works. However is I add another field that has a high cardinality like customer number then the resulting pre-aggregation starts to return wrong results. This seems ot happens because data is missing. I checked the query that is sent to the data source and that is correct. The issue is compounded the more fields I add to the pre-aggregation. It does not mater if they are from the same cube or from a different one.

```
pre_aggregations:
- name: invoices
dimensions:
- CUBE.price_condition_code
- CUBE.sales_rep_code
- product.section
- product.family
- product.product_name
measures:
- CUBE.sales_amount
- CUBE.quantity
refresh_key:
every: 1 day
incremental: true
update_window: 4 weeks
time_dimension: CUBE.invoice_date
granularity: month
partition_granularity: month
type: rollup
external: true
build_range_start:
sql: SELECT CURRENT_DATE - INTERVAL '2' month FROM DUAL
build_range_end:
sql: SELECT CURRENT_DATE FROM DUAL
```

**Version:**
1.6.65 CUBE and 1.6.54 Cubestore

Contributor guide

Open the contributing guide

Research direction

The payload names no repository files or tests; start by reproducing the Cube Core 1.6.65 and Cubestore 1.6.54 setup with Oracle and Minio using the supplied pre-aggregation configuration. Compare results with and without a high-cardinality dimension, then verify the pre-aggregation contains all expected data and matches the source query.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.