apache / apache/datafusion-comet
Cast from float/double to decimal should return NULL for NaN/Infinity under ANSI mode
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
### What is the problem the feature request solves?
The float/double-to-decimal cast is currently marked `Incompatible` in `CometCast` (`spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:409-411, 420-422`), so it only runs with `spark.comet.expression.Cast.allowIncompatible=true`. One of the divergences that keeps it incompatible: under ANSI mode, Comet throws for NaN and +/-Infinity inputs where Spark returns NULL.
Spark 4.1 (`Cast.scala`, both interpreted and codegen paths) wraps `Decimal(fractional.toDouble(b))` in a catch of `NumberFormatException`, so `BigDecimal.valueOf(NaN)` results in NULL even under ANSI. Only finite values that overflow the target precision throw.
Comet (`native/spark-expr/src/conversion_funcs/numeric.rs:907-937`) nulls NaN/Inf via `unary_opt`, but then, under ANSI, rescans for newly introduced nulls and raises `NumericValueOutOfRange { value: "NaN" | "inf" }`. The native unit test `test_cast_float_to_decimal_ansi_nan_errors` (`numeric.rs:1424-1450`) locks in this incorrect behavior.
### Describe the potential solution
Exempt NaN and +/-Infinity from the ANSI overflow rescan in `cast_floating_point_to_decimal128` (only finite overflowing values should error), and invert the native test. This removes one of the blockers to promoting the cast pair out of `Incompatible`.
### Additional context
Found by an ANSI-mode audit of all native expressions against Spark 4.1.1. Filed as an enhancement rather than a bug because the cast pair is already gated behind `allowIncompatible`.
Contributor guide
Research direction
Start with the incompatible cast entries in spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:409-411 and 420-422, then read cast_floating_point_to_decimal128 in native/spark-expr/src/conversion_funcs/numeric.rs:907-937. Update the ANSI overflow handling so NaN and +/-Infinity become NULL while finite overflowing values still error, and invert test_cast_float_to_decimal_ansi_nan_errors at numeric.rs:1424-1450. Done means the native test reflects Spark 4.1 behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, scala, spark
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100