[Bug] pmod and mod report Arrow's divide-by-zero message instead of Spark's REMAINDER_BY_ZERO text
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
Under `datafusion.execution.enable_ansi_mode = true`, `datafusion-spark`'s
`pmod` and `mod` correctly raise on a zero divisor, but the message is Arrow's:
```
DataFusion error: Arrow error: Divide by zero error
```
Spark 4.2.0 raises:
```
[REMAINDER_BY_ZERO] Remainder by zero. Use `try_mod` to tolerate divisor being 0
and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to
bypass this error. SQLSTATE: 22012
```
The condition detected is the same. Only the text differs. DataFusion does not
model Spark's error classes or SQLSTATE values, so an exact reproduction is not
possible, but the message body is expressible.
### Cross-version note
The error class name is itself version dependent:
| Spark | Error class |
| ------ | -------------------- |
| 3.5.8 | `DIVIDE_BY_ZERO` |
| 4.0.4 | `DIVIDE_BY_ZERO` |
| 4.1.3 | `REMAINDER_BY_ZERO` |
| 4.2.0 | `REMAINDER_BY_ZERO` |
`Pmod.eval` calls `QueryExecutionErrors.divideByZeroError` in 3.5.8 and 4.0.4
and `QueryExecutionErrors.remainderByZeroError` in 4.1.3 and later. DataFusion
has no mechanism for version-specific expectations, tracked by
https://github.com/apache/datafusion/issues/23887.
### To Reproduce
```sql
set datafusion.execution.enable_ansi_mode = true;
SELECT pmod(10::int, 0::int);
-- DataFusion error: Arrow error: Divide by zero error
```
### Expected behavior
A message matching Spark 4.2.0's `REMAINDER_BY_ZERO` text.
### Scope
This is a repository-wide question rather than a `pmod` question. The in-repo
precedent, `datafusion/spark/src/function/math/abs.rs` and
`datafusion/sqllogictest/test_files/spark/math/abs.slt`, asserts Arrow's native
overflow message rather than Spark's `ARITHMETIC_OVERFLOW` text, so changing
`pmod` alone would make the crate less consistent. A decision on whether
`datafusion-spark` should reproduce Spark's ANSI error text in general would be
useful before individual functions are changed.
### Relevant code
`datafusion/spark/src/function/math/modulus.rs`, `spark_pmod` and `try_rem`.
Surfaced by the audit-datafusion-spark-expression skill.
Contributor guide
Research direction
Start with datafusion/spark/src/function/math/modulus.rs, especially spark_pmod and try_rem, then compare the precedent in datafusion/spark/src/function/math/abs.rs and datafusion/sqllogictest/test_files/spark/math/abs.slt. Review the cross-version Spark behavior and decide whether ANSI error text should be reproduced consistently across datafusion-spark. Done means the repository-wide policy is settled and the expected scope and tests for this issue are clear.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, spark, sql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100