Native arithmetic does not honor ANSI overflow semantics (e.g. negate of Int.MinValue)
- Dominant language
- Rust
- Stars
- 1.8k
- Forks
- 241
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 21
Description
### Describe the bug
Under ANSI mode (`spark.sql.ansi.enabled=true`, the default in Spark 4.x), the native engine does not honor Spark's arithmetic-overflow semantics. Integer/long negation that overflows wraps around in the native engine, whereas vanilla Spark throws `SparkArithmeticException: [ARITHMETIC_OVERFLOW]`.
### How to reproduce
On Spark 4.x (ANSI on by default):
```sql
CREATE TABLE t1(col1 int) USING parquet;
INSERT INTO t1 VALUES (-2147483648);
SELECT negative(col1), -(col1) FROM t1;
```
- With Auron disabled: throws `[ARITHMETIC_OVERFLOW]`.
- With Auron enabled: returns `-2147483648` (no error).
This surfaced via the `AuronExpressionSuite."UnaryMinus"` test once the config alt-key fix (#2360) made `checkSparkAnswerAndOperator`'s `spark.auron.enable=false` baseline actually run vanilla Spark. As a stopgap, that test now runs with `spark.sql.ansi.enabled=false` so both engines wrap consistently (see PR #2361); this issue tracks the underlying native ANSI-overflow gap.
### Suggested fix
Make the native engine honor ANSI overflow semantics when `spark.sql.ansi.enabled=true`
Contributor guide
Research direction
Start with the AuronExpressionSuite."UnaryMinus" test and the checkSparkAnswerAndOperator path described in the issue, comparing the vanilla Spark and native-engine results under spark.sql.ansi.enabled=true. Verify the behavior with the provided SQL reproduction and ensure native integer and long negation raises SparkArithmeticException for overflow while non-ANSI behavior remains consistent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100