apache / apache/datafusion-comet
Native panic casting to negative-scale decimal when spark.sql.legacy.allowNegativeScaleOfDecimal=true
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
### Describe the bug
With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, casting any integer column to a negative-scale decimal crashes the native engine:
```
Comet native panic: panicked at library/core/src/num/mod.rs:475:5:
attempt to multiply with overflow
```
`spark.sql.legacy.allowNegativeScaleOfDecimal` is one of the configs listed under #4180. `CometCast` does consult it (`CometCast.scala:279-289`), but only for the **decimal → string** direction: a negative-scale target is reported `Compatible()` when the flag is on. Casting *into* a negative-scale decimal, and arithmetic on negative-scale decimals, are not guarded at all.
Note that the native workspace sets `overflow-checks = false` for the release profile (`native/Cargo.toml:67`), so a release build will not panic here. The same multiplication wraps instead, which would turn this into silently wrong values rather than a crash. I verified the panic on a debug build and the profile setting by inspection; I did not run a release build to confirm what the wrapped result looks like.
### Steps to reproduce
Negative scale cannot be written in SQL (the parser rejects `decimal(20,-5)` outright), so this needs the DataFrame API:
```scala
import org.apache.spark.sql.functions.col
import org.apache.spark.sql.types.DecimalType
spark.conf.set("spark.sql.legacy.allowNegativeScaleOfDecimal", "true")
val path = "/tmp/negscale"
spark.sql("SELECT cast(n as bigint) as n FROM VALUES (1),(2),(3) AS v(n)")
.write.mode("overwrite").parquet(path)
val df = spark.read.parquet(path)
spark.conf.set("spark.comet.enabled", "false")
df.select(col("n").cast(DecimalType(10, -1))).collect()
// [0E+1], [0E+1], [0E+1] (Spark rounds to the negative scale)
spark.conf.set("spark.comet.enabled", "true")
df.select(col("n").cast(DecimalType(10, -1))).collect()
// org.apache.comet.CometNativeException: native panic: attempt to multiply with overflow
```
Also reproduced with:
* `DecimalType(20, -5)` — same panic;
* arithmetic on the result, `n + n` / `n * 2` — `attempt to subtract with overflow`;
* casting the negative-scale decimal back to string — same panic, i.e. the one direction `CometCast` does guard is unreachable because producing the value crashes first.
With `spark.sql.legacy.allowNegativeScaleOfDecimal=false` both engines raise `NEGATIVE_SCALE_DISALLOWED` identically, so only the `true` case is affected.
### Expected behavior
Comet should either handle negative-scale decimals correctly, or report `Unsupported`/`Incompatible` for any cast or arithmetic involving a negative-scale decimal so the expression falls back to Spark. It should not panic, and it should not wrap silently in release builds.
### Additional context
* Reproduced on `apache/main` @ `0761e549a`, default Maven profile (Spark 4.1.2), macOS aarch64, debug build.
* Found while auditing the config inventory in #4180.
Contributor guide
Assessment
This issue has not been assessed yet.