[Bug] spark factorial rejects STRING, DECIMAL and floating-point inputs that Spark implicitly casts to INT
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
Spark's `Factorial` extends `ImplicitCastInputTypes` with `inputTypes = Seq(IntegerType)`. Under
ANSI mode the implicit cast rule (`AnsiTypeCoercion.implicitCast` -> `Cast.canANSIStoreAssign`)
allows any numeric type to be cast to INT and any string to be cast to an atomic type; the legacy
rule allows the same. So Spark accepts DECIMAL, FLOAT, DOUBLE and STRING arguments. DataFusion
rejects all of them at planning time.
### To Reproduce
Spark SQL, `pyspark==4.2.0`, both ANSI settings unless noted:
```sql
SELECT factorial(CAST(5 AS DECIMAL(10,0))); -- 120
SELECT factorial(5.0); -- 120
SELECT factorial(CAST(5.7 AS DOUBLE)); -- 120 (fraction truncated)
SELECT factorial(CAST(20.9 AS FLOAT)); -- 2432902008176640000
SELECT factorial('5'); -- 120
SELECT factorial(' 5 '); -- 120
SELECT factorial('5.0'); -- ANSI: [CAST_INVALID_INPUT] error; non-ANSI: 120
SELECT factorial('abc'); -- ANSI: [CAST_INVALID_INPUT] error; non-ANSI: NULL
SELECT factorial(''); -- ANSI: [CAST_INVALID_INPUT] error; non-ANSI: NULL
SELECT factorial(CAST(99999999999 AS DECIMAL(20,0))); -- ANSI: [CAST_OVERFLOW] error; non-ANSI: NULL
```
DataFusion (`datafusion-cli --spark`): on `main` every one of these fails with
`Failed to coerce arguments to satisfy a call to 'factorial' function: coercion from Utf8 to the
signature Exact(Int32) failed` (and likewise for Float64 / Decimal128). With the integer-width fix
for #24940 applied they still fail, now with
`Function 'factorial' requires Int32, but received String (DataType: Utf8)` (and likewise
`Float64` and `Decimal(10, 0)`), so this divergence survives that fix.
### Expected behavior
Match Spark: accept the types Spark accepts and apply Spark's cast semantics, including the
ANSI-dependent behaviour for malformed strings and overflow. (DataFusion currently fails at the
cast in both modes, since the function does not consult `datafusion.execution.enable_ansi_mode`.)
### Additional context
Not fixed together with the integer-width fix because it needs a decision, not a mechanical change:
1. #23889 is the same `ImplicitCastInputTypes` gap for `next_day`; its discussion raises the general
question of whether `datafusion-spark` should model Spark's implicit casts, and a general
approach should probably cover both.
2. The string and overflow cases are ANSI dependent in Spark, so any widening has to route through
a Spark-compatible cast rather than `arrow::compute::cast`, which errors in both modes.
Relevant code: `datafusion/spark/src/function/math/factorial.rs`, `SparkFactorial::new`.
Surfaced by the `audit-datafusion-spark-expression` skill.
Contributor guide
Research direction
Start in datafusion/spark/src/function/math/factorial.rs at SparkFactorial::new, then read the implicit-cast discussion in #23889. Determine a Spark-compatible casting approach that respects enable_ansi_mode; done means DECIMAL, floating-point, and STRING inputs match Spark's accepted values, malformed-string behavior, and overflow behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, spark, sql
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100