apache / apache/auron

Native arithmetic does not honor ANSI overflow semantics (e.g. negate of Int.MinValue)

Open
#2,368 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 12h
Merged PRs (30d)
21

Description

### Describe the bug

Under ANSI mode (`spark.sql.ansi.enabled=true`, the default in Spark 4.x), the native engine does not honor Spark's arithmetic-overflow semantics. Integer/long negation that overflows wraps around in the native engine, whereas vanilla Spark throws `SparkArithmeticException: [ARITHMETIC_OVERFLOW]`.

### How to reproduce

On Spark 4.x (ANSI on by default):

```sql
CREATE TABLE t1(col1 int) USING parquet;
INSERT INTO t1 VALUES (-2147483648);
SELECT negative(col1), -(col1) FROM t1;
```

- With Auron disabled: throws `[ARITHMETIC_OVERFLOW]`.
- With Auron enabled: returns `-2147483648` (no error).

This surfaced via the `AuronExpressionSuite."UnaryMinus"` test once the config alt-key fix (#2360) made `checkSparkAnswerAndOperator`'s `spark.auron.enable=false` baseline actually run vanilla Spark. As a stopgap, that test now runs with `spark.sql.ansi.enabled=false` so both engines wrap consistently (see PR #2361); this issue tracks the underlying native ANSI-overflow gap.

### Suggested fix

Make the native engine honor ANSI overflow semantics when `spark.sql.ansi.enabled=true`

Contributor guide

Open the contributing guide

Research direction

Start with the AuronExpressionSuite."UnaryMinus" test and the checkSparkAnswerAndOperator path described in the issue, comparing the vanilla Spark and native-engine results under spark.sql.ansi.enabled=true. Verify the behavior with the provided SQL reproduction and ensure native integer and long negation raises SparkArithmeticException for overflow while non-ANSI behavior remains consistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.