[v1.6] Decimal precision mismatch in CubeStore pre-aggregations due to DataFusion's SUM precision increase
- Dominant language
- Rust
- Stars
- 20.8k
- Forks
- 2.1k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 203
Description
# Decimal precision mismatch in CubeStore pre-aggregations due to DataFusion's SUM precision increase
## Describe the bug
When building pre-aggregations, CubeStore fails with a decimal precision mismatch error. This occurs because DataFusion automatically increases decimal precision by 10 during `SUM()` operations to prevent overflow, but the table schema doesn't account for this increase.
**Error:**
```
2026-02-04T10:05:47.528Z ERROR [cubestore::cluster::ingestion::job_runner] Error while running job 6: Internal: task 12447 panicked with message "called `Result::unwrap()` on an `Err` value: InvalidArgumentError(\"column types must match schema types, expected Decimal128(18, 5) but found Decimal128(28, 5) at column index 11\")"
```
## Root Cause: DataFusion's Decimal Arithmetic
Starting with Cube 1.6 (which upgraded DataFusion), the query engine follows standard SQL decimal arithmetic rules to prevent overflow during aggregations.
### SUM() Precision Formula
When DataFusion performs a `SUM()` on a decimal column, it calculates the result type as:
```
Result precision = min(MAX_PRECISION, input_precision + 10)
Result scale = input_scale (unchanged)
```
https://github.com/cube-js/arrow-datafusion/blob/6a8b3a0e1fa36428d8e17a32a46b5808d04065ec/datafusion/functions-aggregate/src/sum.rs#L161
```rust
DataType::Decimal128(precision, scale) => {
// in the spark, the result type is DECIMAL(min(38,precision+10), s)
// ref: https://github.com/apache/spark/blob/fcf636d9eb8d645c24be3db2d599aba2d7e2955a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Sum.scala#L66
let new_precision = DECIMAL128_MAX_PRECISION.min(*precision + 10);
Ok(DataType::Decimal128(new_precision, *scale))
}
````
Contributor guide
Research direction
Start with the DataFusion SUM implementation referenced in the issue, then trace CubeStore's pre-aggregation schema and ingestion path to find where the Decimal128 result type is assumed. Reproduce a pre-aggregation containing SUM on a decimal column and verify that the build completes without a column-schema precision mismatch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100