apache / apache/datafusion-comet
Wide-decimal overflow reports a different value than Spark
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### Describe the bug
In ANSI mode, Comet's wide-decimal arithmetic overflow reports a different `value` parameter from Spark. PR #5169 fixes the exception type and propagates the error class, SQLSTATE, and query context, but Comet still formats the rescaled i256 result while Spark formats its pre-`toPrecision` Decimal.
This is a follow-up to #5072.
### Steps to reproduce
Using Spark 4.1.2, run the query once with Comet disabled and once with Comet enabled:
```sql
SET spark.sql.ansi.enabled = true;
CREATE TABLE tbl (_1 DECIMAL(20, 0)) USING PARQUET;
INSERT INTO tbl VALUES (11000000000000000000);
SELECT _1 * _1 FROM tbl;
```
Spark reports:
```text
[NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION] 121000000000000000000000000000000000000 cannot be represented as Decimal(38, 6). If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error, and return NULL instead. SQLSTATE: 22003
```
Comet reports:
```text
[NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION] 121000000000000000000000000000000000000.000000 cannot be represented as Decimal(38, 6). If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error, and return NULL instead. SQLSTATE: 22003
```
### Expected behavior
Comet should match Spark's `value` parameter and report `121000000000000000000000000000000000000` without the output-scale suffix.
### Additional context
The regression in `spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala` added by #5169 documents the current difference: Spark formats its pre-`toPrecision` Decimal, while Comet formats the rescaled i256 value. The test intentionally compares the structured error fields and query context without comparing this differing message parameter.
Matching Spark is not only a scale-formatting change. Spark's `Decimal.$times` multiplies with `MATH_CONTEXT`, which is `MathContext(38, HALF_UP)`, so the value in Spark's error has already been rounded to 38 significant digits before `toPrecision` is attempted. The reproduction above rounds exactly because `1.21e38` has trailing zeros; products with non-zero low digits can differ from Comet's exact i256 even after natural-scale formatting. A fix must reproduce this significant-digit rounding as well as the scale formatting.
Contributor guide
Assessment
This issue has not been assessed yet.