cube-js / cube-js/cube

[v1.6] Decimal precision mismatch in CubeStore pre-aggregations due to DataFusion's SUM precision increase

Open
#10,365 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 10h
Merged PRs (30d)
203

Description

# Decimal precision mismatch in CubeStore pre-aggregations due to DataFusion's SUM precision increase

## Describe the bug

When building pre-aggregations, CubeStore fails with a decimal precision mismatch error. This occurs because DataFusion automatically increases decimal precision by 10 during `SUM()` operations to prevent overflow, but the table schema doesn't account for this increase.

**Error:**
```
2026-02-04T10:05:47.528Z ERROR [cubestore::cluster::ingestion::job_runner] Error while running job 6: Internal: task 12447 panicked with message "called `Result::unwrap()` on an `Err` value: InvalidArgumentError(\"column types must match schema types, expected Decimal128(18, 5) but found Decimal128(28, 5) at column index 11\")"
```

## Root Cause: DataFusion's Decimal Arithmetic

Starting with Cube 1.6 (which upgraded DataFusion), the query engine follows standard SQL decimal arithmetic rules to prevent overflow during aggregations.

### SUM() Precision Formula

When DataFusion performs a `SUM()` on a decimal column, it calculates the result type as:

```
Result precision = min(MAX_PRECISION, input_precision + 10)
Result scale = input_scale (unchanged)
```
https://github.com/cube-js/arrow-datafusion/blob/6a8b3a0e1fa36428d8e17a32a46b5808d04065ec/datafusion/functions-aggregate/src/sum.rs#L161

```rust
DataType::Decimal128(precision, scale) => {
// in the spark, the result type is DECIMAL(min(38,precision+10), s)
// ref: https://github.com/apache/spark/blob/fcf636d9eb8d645c24be3db2d599aba2d7e2955a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Sum.scala#L66
let new_precision = DECIMAL128_MAX_PRECISION.min(*precision + 10);
Ok(DataType::Decimal128(new_precision, *scale))
}
````

Contributor guide

Open the contributing guide

Research direction

Start with the DataFusion SUM implementation referenced in the issue, then trace CubeStore's pre-aggregation schema and ingestion path to find where the Decimal128 result type is assumed. Reproduce a pre-aggregation containing SUM on a decimal column and verify that the build completes without a column-schema precision mismatch.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.