apache / apache/datafusion-comet

Wide-decimal overflow reports a different value than Spark

Open
#5,211 1 comment 1 reaction 1 assignee Claimed by @sam-1112 View on GitHub
good first issue priority:medium
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### Describe the bug

In ANSI mode, Comet's wide-decimal arithmetic overflow reports a different `value` parameter from Spark. PR #5169 fixes the exception type and propagates the error class, SQLSTATE, and query context, but Comet still formats the rescaled i256 result while Spark formats its pre-`toPrecision` Decimal.

This is a follow-up to #5072.

### Steps to reproduce

Using Spark 4.1.2, run the query once with Comet disabled and once with Comet enabled:

```sql
SET spark.sql.ansi.enabled = true;

CREATE TABLE tbl (_1 DECIMAL(20, 0)) USING PARQUET;
INSERT INTO tbl VALUES (11000000000000000000);

SELECT _1 * _1 FROM tbl;
```

Spark reports:

```text
[NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION] 121000000000000000000000000000000000000 cannot be represented as Decimal(38, 6). If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error, and return NULL instead. SQLSTATE: 22003
```

Comet reports:

```text
[NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION] 121000000000000000000000000000000000000.000000 cannot be represented as Decimal(38, 6). If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error, and return NULL instead. SQLSTATE: 22003
```

### Expected behavior

Comet should match Spark's `value` parameter and report `121000000000000000000000000000000000000` without the output-scale suffix.

### Additional context

The regression in `spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala` added by #5169 documents the current difference: Spark formats its pre-`toPrecision` Decimal, while Comet formats the rescaled i256 value. The test intentionally compares the structured error fields and query context without comparing this differing message parameter.

Matching Spark is not only a scale-formatting change. Spark's `Decimal.$times` multiplies with `MATH_CONTEXT`, which is `MathContext(38, HALF_UP)`, so the value in Spark's error has already been rounded to 38 significant digits before `toPrecision` is attempted. The reproduction above rounds exactly because `1.21e38` has trailing zeros; products with non-zero low digits can differ from Comet's exact i256 even after natural-scale formatting. A fix must reproduce this significant-digit rounding as well as the scale formatting.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.