Pre-Aggregations missing data
- Dominant language
- Rust
- Stars
- 20.8k
- Forks
- 2.1k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 181
Description
I am running Cube Core with Oracle as a source database and Minio for pre-aggregation storage. I am running into an issue that causes pre-aggregations to not contain all data that they should. It seems ot be driven by the size of the partition.
Below is an example of a pre.aggregation. The below example works. However is I add another field that has a high cardinality like customer number then the resulting pre-aggregation starts to return wrong results. This seems ot happens because data is missing. I checked the query that is sent to the data source and that is correct. The issue is compounded the more fields I add to the pre-aggregation. It does not mater if they are from the same cube or from a different one.
```
pre_aggregations:
- name: invoices
dimensions:
- CUBE.price_condition_code
- CUBE.sales_rep_code
- product.section
- product.family
- product.product_name
measures:
- CUBE.sales_amount
- CUBE.quantity
refresh_key:
every: 1 day
incremental: true
update_window: 4 weeks
time_dimension: CUBE.invoice_date
granularity: month
partition_granularity: month
type: rollup
external: true
build_range_start:
sql: SELECT CURRENT_DATE - INTERVAL '2' month FROM DUAL
build_range_end:
sql: SELECT CURRENT_DATE FROM DUAL
```
**Version:**
1.6.65 CUBE and 1.6.54 Cubestore
Contributor guide
Research direction
The payload names no repository files or tests; start by reproducing the Cube Core 1.6.65 and Cubestore 1.6.54 setup with Oracle and Minio using the supplied pre-aggregation configuration. Compare results with and without a high-cardinality dimension, then verify the pre-aggregation contains all expected data and matches the source query.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100