[Bug] spark factorial: BIGINT overflow should wrap in non-ANSI mode and raise CAST_OVERFLOW in ANSI mode, as in Spark
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
With the integer-width fix in #24943 applied, `datafusion-spark` accepts BIGINT
arguments to `factorial` by implicitly casting them to INT. That cast raises an
Arrow overflow error regardless of `datafusion.execution.enable_ansi_mode`.
Spark 4.2.0 raises `CAST_OVERFLOW` in ANSI mode. In non-ANSI mode, its
BIGINT-to-INT cast wraps to a signed 32-bit integer, and `factorial` evaluates
that wrapped value. The result can be NULL or a valid factorial.
This is a follow-up to #24940 / #24943. Reproduced against PR #24943 at
`ff3968e66e9cef4dee8c034b568386505a6590fb`. The PR is still open as of
2026-09-04. Without that fix, BIGINT arguments are rejected during type coercion.
### To Reproduce
DataFusion with Spark functions registered and #24943 applied:
```sql
SET datafusion.execution.enable_ansi_mode = false;
SELECT factorial(CAST(5000000000 AS BIGINT));
-- Arrow error: Cast error: Can't cast value 5000000000 to type Int32
SELECT factorial(CAST(4294967301 AS BIGINT));
-- Arrow error: Cast error: Can't cast value 4294967301 to type Int32
```
Running either SELECT with `datafusion.execution.enable_ansi_mode = true`
also raises the corresponding Arrow cast error. For these constant expressions,
the error is reported by the `simplify_expressions` optimizer rule.
Spark SQL, verified with a live Spark 4.2.0 session (`pyspark==4.2.0`):
```sql
SET spark.sql.ansi.enabled = false;
SELECT factorial(CAST(5000000000 AS BIGINT)); -- NULL
SELECT factorial(CAST(4294967301 AS BIGINT)); -- 120
SET spark.sql.ansi.enabled = true;
SELECT factorial(CAST(5000000000 AS BIGINT)); -- [CAST_OVERFLOW]
SELECT factorial(CAST(4294967301 AS BIGINT)); -- [CAST_OVERFLOW]
```
In non-ANSI mode, `5000000000` wraps to `705032704`, which is outside
`factorial`'s supported range of 0 through 20. `4294967301` wraps to `5`, so
its factorial is `120`. Returning NULL for every overflowing BIGINT would
therefore still diverge from Spark.
### Expected behavior
Match Spark's implicit BIGINT-to-INT cast semantics: with
`datafusion.execution.enable_ansi_mode = false`, wrap to signed INT and evaluate
`factorial` on that value. With ANSI mode enabled, raise an overflow error.
### Additional context
`SparkFactorial::new` in `datafusion/spark/src/function/math/factorial.rs`
uses `Signature::coercible` after #24943. Type coercion inserts the cast before
the function runs. Reading the ANSI setting inside `factorial` alone cannot
change a cast that has already failed. The implementation needs to account for
Spark's cast semantics at that earlier stage.
The latest #24943 diff has no out-of-range BIGINT `query error` case in
`datafusion/sqllogictest/test_files/spark/math/factorial.slt`. Regression coverage
should include both ANSI modes and an overflowing value that wraps into 0..20,
for scalar and column inputs.
Related to #24941, which covers additional accepted input types and their
ANSI-dependent casts. Part of #23929.
Surfaced by the audit-datafusion-spark-expression skill.
Contributor guide
Research direction
Start in datafusion/spark/src/function/math/factorial.rs and inspect how Signature::coercible inserts casts before factorial. Then read datafusion/sqllogictest/test_files/spark/math/factorial.slt and run its existing cases. Done means covering scalar and column BIGINT inputs in both ANSI modes, including values that wrap into 0..20, with Spark-compatible results or overflow errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 65/100