apache / apache/datafusion

[Bug] pmod rejects string arguments that Spark implicitly casts to a numeric type

Open
#23,896 1 comment 0 reactions 1 assignee Claimed by @amitvijapur View on GitHub
bug spark
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Spark's `Pmod` declares `inputType = NumericType` and relies on the analyzer's
string promotion rule, so a string argument is implicitly cast. DataFusion's
`SparkPmod` uses `Signature::numeric(2, Volatility::Immutable)`, which rejects
strings at planning time.

### To Reproduce

Spark 4.2.0 (verified with `pyspark==4.2.0`), and covered by Spark's own golden
file `sql/core/src/test/resources/sql-tests/results/typeCoercion/native/promoteStrings.sql.out`:

```sql
SELECT pmod('1', CAST(1 AS TINYINT)); -- 0, result type bigint
SELECT pmod('10', CAST(3 AS INT)); -- ansi on: 1 (int); ansi off: 1.0 (double)
SELECT pmod('1', '1'); -- DATATYPE_MISMATCH.BINARY_OP_WRONG_TYPE
```

Note that Spark's promotion target differs by mode: with
`spark.sql.ansi.enabled = true` the string is cast to the other argument's type,
and with it disabled the pair is promoted to `double`. Two strings are rejected
in both modes.

DataFusion:

```sql
SELECT pmod('10'::string, 3::int);
-- Error during planning: For function 'pmod' Utf8 and Int32 are not coercible
-- to a common numeric type.
```

### Expected behavior

A string argument is accepted and coerced the way Spark coerces it.

### Scope

This is not specific to `pmod`. Every `datafusion-spark` function that uses
`Signature::numeric` inherits it, and reproducing Spark's mode-dependent
promotion target needs a decision about how far `datafusion-spark` should go in
emulating Spark's analyzer. Filing it so the gap is visible rather than
proposing a `pmod`-local workaround.

A commented-out query recording the Spark 4.2.0 result is in
`datafusion/sqllogictest/test_files/spark/math/pmod.slt`.

### Relevant code

`datafusion/spark/src/function/math/modulus.rs`, `SparkPmod::new`.

Surfaced by the audit-datafusion-spark-expression skill.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.