apache / apache/datafusion-comet

Native panic casting to negative-scale decimal when spark.sql.legacy.allowNegativeScaleOfDecimal=true

Open
#5,013 3 comments 0 reactions 2 assignees Claimed by @comphead View on GitHub
area:expressions bug crash priority:high
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

### Describe the bug

With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, casting any integer column to a negative-scale decimal crashes the native engine:

```
Comet native panic: panicked at library/core/src/num/mod.rs:475:5:
attempt to multiply with overflow
```

`spark.sql.legacy.allowNegativeScaleOfDecimal` is one of the configs listed under #4180. `CometCast` does consult it (`CometCast.scala:279-289`), but only for the **decimal → string** direction: a negative-scale target is reported `Compatible()` when the flag is on. Casting *into* a negative-scale decimal, and arithmetic on negative-scale decimals, are not guarded at all.

Note that the native workspace sets `overflow-checks = false` for the release profile (`native/Cargo.toml:67`), so a release build will not panic here. The same multiplication wraps instead, which would turn this into silently wrong values rather than a crash. I verified the panic on a debug build and the profile setting by inspection; I did not run a release build to confirm what the wrapped result looks like.

### Steps to reproduce

Negative scale cannot be written in SQL (the parser rejects `decimal(20,-5)` outright), so this needs the DataFrame API:

```scala
import org.apache.spark.sql.functions.col
import org.apache.spark.sql.types.DecimalType

spark.conf.set("spark.sql.legacy.allowNegativeScaleOfDecimal", "true")

val path = "/tmp/negscale"
spark.sql("SELECT cast(n as bigint) as n FROM VALUES (1),(2),(3) AS v(n)")
.write.mode("overwrite").parquet(path)
val df = spark.read.parquet(path)

spark.conf.set("spark.comet.enabled", "false")
df.select(col("n").cast(DecimalType(10, -1))).collect()
// [0E+1], [0E+1], [0E+1] (Spark rounds to the negative scale)

spark.conf.set("spark.comet.enabled", "true")
df.select(col("n").cast(DecimalType(10, -1))).collect()
// org.apache.comet.CometNativeException: native panic: attempt to multiply with overflow
```

Also reproduced with:

* `DecimalType(20, -5)` — same panic;
* arithmetic on the result, `n + n` / `n * 2` — `attempt to subtract with overflow`;
* casting the negative-scale decimal back to string — same panic, i.e. the one direction `CometCast` does guard is unreachable because producing the value crashes first.

With `spark.sql.legacy.allowNegativeScaleOfDecimal=false` both engines raise `NEGATIVE_SCALE_DISALLOWED` identically, so only the `true` case is affected.

### Expected behavior

Comet should either handle negative-scale decimals correctly, or report `Unsupported`/`Incompatible` for any cast or arithmetic involving a negative-scale decimal so the expression falls back to Spark. It should not panic, and it should not wrap silently in release builds.

### Additional context

* Reproduced on `apache/main` @ `0761e549a`, default Maven profile (Spark 4.1.2), macOS aarch64, debug build.
* Found while auditing the config inventory in #4180.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.