apache / apache/datafusion

[Bug] spark factorial rejects STRING, DECIMAL and floating-point inputs that Spark implicitly casts to INT

Open
#24,941 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Spark's `Factorial` extends `ImplicitCastInputTypes` with `inputTypes = Seq(IntegerType)`. Under
ANSI mode the implicit cast rule (`AnsiTypeCoercion.implicitCast` -> `Cast.canANSIStoreAssign`)
allows any numeric type to be cast to INT and any string to be cast to an atomic type; the legacy
rule allows the same. So Spark accepts DECIMAL, FLOAT, DOUBLE and STRING arguments. DataFusion
rejects all of them at planning time.

### To Reproduce

Spark SQL, `pyspark==4.2.0`, both ANSI settings unless noted:

```sql
SELECT factorial(CAST(5 AS DECIMAL(10,0))); -- 120
SELECT factorial(5.0); -- 120
SELECT factorial(CAST(5.7 AS DOUBLE)); -- 120 (fraction truncated)
SELECT factorial(CAST(20.9 AS FLOAT)); -- 2432902008176640000
SELECT factorial('5'); -- 120
SELECT factorial(' 5 '); -- 120
SELECT factorial('5.0'); -- ANSI: [CAST_INVALID_INPUT] error; non-ANSI: 120
SELECT factorial('abc'); -- ANSI: [CAST_INVALID_INPUT] error; non-ANSI: NULL
SELECT factorial(''); -- ANSI: [CAST_INVALID_INPUT] error; non-ANSI: NULL
SELECT factorial(CAST(99999999999 AS DECIMAL(20,0))); -- ANSI: [CAST_OVERFLOW] error; non-ANSI: NULL
```

DataFusion (`datafusion-cli --spark`): on `main` every one of these fails with
`Failed to coerce arguments to satisfy a call to 'factorial' function: coercion from Utf8 to the
signature Exact(Int32) failed` (and likewise for Float64 / Decimal128). With the integer-width fix
for #24940 applied they still fail, now with
`Function 'factorial' requires Int32, but received String (DataType: Utf8)` (and likewise
`Float64` and `Decimal(10, 0)`), so this divergence survives that fix.

### Expected behavior

Match Spark: accept the types Spark accepts and apply Spark's cast semantics, including the
ANSI-dependent behaviour for malformed strings and overflow. (DataFusion currently fails at the
cast in both modes, since the function does not consult `datafusion.execution.enable_ansi_mode`.)

### Additional context

Not fixed together with the integer-width fix because it needs a decision, not a mechanical change:

1. #23889 is the same `ImplicitCastInputTypes` gap for `next_day`; its discussion raises the general
question of whether `datafusion-spark` should model Spark's implicit casts, and a general
approach should probably cover both.
2. The string and overflow cases are ANSI dependent in Spark, so any widening has to route through
a Spark-compatible cast rather than `arrow::compute::cast`, which errors in both modes.

Relevant code: `datafusion/spark/src/function/math/factorial.rs`, `SparkFactorial::new`.
Surfaced by the `audit-datafusion-spark-expression` skill.

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/spark/src/function/math/factorial.rs at SparkFactorial::new, then read the implicit-cast discussion in #23889. Determine a Spark-compatible casting approach that respects enable_ansi_mode; done means DECIMAL, floating-point, and STRING inputs match Spark's accepted values, malformed-string behavior, and overflow behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, spark, sql
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.