apache / apache/datafusion-comet

Cast from float/double to decimal should return NULL for NaN/Infinity under ANSI mode

Open
#5,069 2 comments 0 reactions 0 assignees View on GitHub
area:expressions enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

### What is the problem the feature request solves?

The float/double-to-decimal cast is currently marked `Incompatible` in `CometCast` (`spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:409-411, 420-422`), so it only runs with `spark.comet.expression.Cast.allowIncompatible=true`. One of the divergences that keeps it incompatible: under ANSI mode, Comet throws for NaN and +/-Infinity inputs where Spark returns NULL.

Spark 4.1 (`Cast.scala`, both interpreted and codegen paths) wraps `Decimal(fractional.toDouble(b))` in a catch of `NumberFormatException`, so `BigDecimal.valueOf(NaN)` results in NULL even under ANSI. Only finite values that overflow the target precision throw.

Comet (`native/spark-expr/src/conversion_funcs/numeric.rs:907-937`) nulls NaN/Inf via `unary_opt`, but then, under ANSI, rescans for newly introduced nulls and raises `NumericValueOutOfRange { value: "NaN" | "inf" }`. The native unit test `test_cast_float_to_decimal_ansi_nan_errors` (`numeric.rs:1424-1450`) locks in this incorrect behavior.

### Describe the potential solution

Exempt NaN and +/-Infinity from the ANSI overflow rescan in `cast_floating_point_to_decimal128` (only finite overflowing values should error), and invert the native test. This removes one of the blockers to promoting the cast pair out of `Incompatible`.

### Additional context

Found by an ANSI-mode audit of all native expressions against Spark 4.1.1. Filed as an enhancement rather than a bug because the cast pair is already gated behind `allowIncompatible`.

Contributor guide

Open the contributing guide

Research direction

Start with the incompatible cast entries in spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:409-411 and 420-422, then read cast_floating_point_to_decimal128 in native/spark-expr/src/conversion_funcs/numeric.rs:907-937. Update the ANSI overflow handling so NaN and +/-Infinity become NULL while finite overflowing values still error, and invert test_cast_float_to_decimal_ansi_nan_errors at numeric.rs:1424-1450. Done means the native test reflects Spark 4.1 behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, scala, spark
Domain
backend
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.