apache / apache/datafusion-comet

Native ANSI errors raised as Arrow errors bypass SparkError conversion (wide decimal, decimal divide, decimal-to-decimal cast)

Open
#5,072 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:expressions bug correctness priority:medium
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### Describe the bug

Three native ANSI error paths raise raw Arrow errors instead of `SparkError`, so they bypass the JSON error converter (`SparkErrorConverter.convertToSparkException` bails when the message is not JSON) and surface as generic `CometNativeException` / `CometQueryExecutionException` instead of `SparkArithmeticException` with the proper error class, SQLSTATE, and query context.

1. **Wide-decimal arithmetic overflow under ANSI**: `native/spark-expr/src/wide_decimal_binary_expr.rs:343` returns `ArrowError::ComputeError("Arithmetic overflow")`. Note that `native/core/src/execution/planner.rs:519-526` deliberately skips the wrapping `CheckOverflow` (which would produce the correct `NUMERIC_VALUE_OUT_OF_RANGE`) when the child is a `WideDecimalBinaryExpr`, so this Arrow error is the only signal the user gets.
2. **Decimal divide/remainder by zero under ANSI**: `native/spark-expr/src/math_funcs/div.rs:96, 123` raises `ArrowError::ComputeError(divide_by_zero_error().to_string())`. The message text contains `[DIVIDE_BY_ZERO]` but the exception type is wrong. The existing test (`CometExpressionSuite.scala:2984-2988`) only asserts on message text, so it passes.
3. **Decimal-to-decimal cast overflow under ANSI**: no dedicated arm in `cast_array`; the fall-through DataFusion cast with `safe=false` propagates `ArrowError::InvalidArgumentError("... too large to store in a Decimal128 of precision ...")` instead of Spark's `[NUMERIC_VALUE_OUT_OF_RANGE.WITHOUT_SUGGESTION] ... cannot be represented as Decimal(p, s)`. This divergence is explicitly special-cased in `CometCastSuite.scala:2402-2405`; fixing it allows deleting that special case.

### Steps to reproduce

```sql
SET spark.sql.ansi.enabled=true;
-- decimal div by zero (correct message text, wrong exception type):
SELECT CAST(1 AS DECIMAL(10,2)) / CAST(0 AS DECIMAL(10,2));
-- decimal to decimal overflow (wrong class and message):
SELECT CAST(CAST(12345.678 AS DECIMAL(20,3)) AS DECIMAL(4,3));
```

### Expected behavior

All three paths raise `SparkError` variants (`NumericValueOutOfRange`, `DivideByZero`) so the converter produces genuine Spark exceptions, as already happens for e.g. integer overflow and array-index errors.

### Additional context

Found by an ANSI-mode audit of all native expressions against Spark 4.1.1.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.