apache / apache/datafusion

[Bug] pmod and mod report Arrow's divide-by-zero message instead of Spark's REMAINDER_BY_ZERO text

Open
#23,897 0 comments 0 reactions 0 assignees View on GitHub
bug spark
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Under `datafusion.execution.enable_ansi_mode = true`, `datafusion-spark`'s
`pmod` and `mod` correctly raise on a zero divisor, but the message is Arrow's:

```
DataFusion error: Arrow error: Divide by zero error
```

Spark 4.2.0 raises:

```
[REMAINDER_BY_ZERO] Remainder by zero. Use `try_mod` to tolerate divisor being 0
and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to
bypass this error. SQLSTATE: 22012
```

The condition detected is the same. Only the text differs. DataFusion does not
model Spark's error classes or SQLSTATE values, so an exact reproduction is not
possible, but the message body is expressible.

### Cross-version note

The error class name is itself version dependent:

| Spark | Error class |
| ------ | -------------------- |
| 3.5.8 | `DIVIDE_BY_ZERO` |
| 4.0.4 | `DIVIDE_BY_ZERO` |
| 4.1.3 | `REMAINDER_BY_ZERO` |
| 4.2.0 | `REMAINDER_BY_ZERO` |

`Pmod.eval` calls `QueryExecutionErrors.divideByZeroError` in 3.5.8 and 4.0.4
and `QueryExecutionErrors.remainderByZeroError` in 4.1.3 and later. DataFusion
has no mechanism for version-specific expectations, tracked by
https://github.com/apache/datafusion/issues/23887.

### To Reproduce

```sql
set datafusion.execution.enable_ansi_mode = true;
SELECT pmod(10::int, 0::int);
-- DataFusion error: Arrow error: Divide by zero error
```

### Expected behavior

A message matching Spark 4.2.0's `REMAINDER_BY_ZERO` text.

### Scope

This is a repository-wide question rather than a `pmod` question. The in-repo
precedent, `datafusion/spark/src/function/math/abs.rs` and
`datafusion/sqllogictest/test_files/spark/math/abs.slt`, asserts Arrow's native
overflow message rather than Spark's `ARITHMETIC_OVERFLOW` text, so changing
`pmod` alone would make the crate less consistent. A decision on whether
`datafusion-spark` should reproduce Spark's ANSI error text in general would be
useful before individual functions are changed.

### Relevant code

`datafusion/spark/src/function/math/modulus.rs`, `spark_pmod` and `try_rem`.

Surfaced by the audit-datafusion-spark-expression skill.

Contributor guide

Open the contributing guide

Research direction

Start with datafusion/spark/src/function/math/modulus.rs, especially spark_pmod and try_rem, then compare the precedent in datafusion/spark/src/function/math/abs.rs and datafusion/sqllogictest/test_files/spark/math/abs.slt. Review the cross-version Spark behavior and decide whether ANSI error text should be reproduced consistently across datafusion-spark. Done means the repository-wide policy is settled and the expected scope and tests for this issue are clear.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, spark, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.