apache / apache/auron

Native float ordering diverges from Spark for NaN and signed zero

Open
#2,517 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 14h
Merged PRs (30d)
22

Description

**Describe the bug**
Native floating-point comparisons and min/max aggregations diverge from Spark SQL semantics for NaN and signed zero (-0.0 vs 0.0).

**To Reproduce**
Run the following DataFrame queries:

```scala
val maxDouble = Seq(Double.NaN, Double.PositiveInfinity, Double.MaxValue)
.map(Tuple1.apply).toDF("a").selectExpr("max(a)").first()
assert(java.lang.Double.isNaN(maxDouble.getDouble(0)))

val comparison = Seq((0.0, -0.0)).toDF("pos", "neg")
.select($"pos" > $"neg").first()
assert(comparison.getBoolean(0) == false)
```

Actual Auron results:
- Native `max(a)` returns a non-NaN value (`Double.PositiveInfinity`) instead of `NaN`.
- Native `0.0 > -0.0` evaluates to `true`.

Reproduced on Spark 3.1 and Spark 3.5.

**Expected behavior**
Spark SQL floating-point ordering semantics:
- `NaN` is equal to `NaN` and greater than any non-NaN value.
- `-0.0` is equal to `0.0`.

`max(a)` should return `NaN` when `NaN` is present, and `0.0 > -0.0` should return `false`.

**Screenshots**
N/A

**Additional context**
- Affected upstream commit: master at `9f90eacb`
- Upstream source file and line references:
- `spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeConverters.scala:576-581`
- `native-engine/datafusion-ext-plans/src/agg/maxmin.rs:110-124, 207-220`
- Distinction from apache/auron#1015 and apache/auron#1016: those changes added `NormalizeNaNAndZero` for grouping keys, but direct comparisons and native min/max aggregations still use native float comparisons directly.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the NaN and signed-zero cases from the issue in Spark and Auron. Read spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeConverters.scala:576-581 and native-engine/datafusion-ext-plans/src/agg/maxmin.rs:110-124, 207-220, then trace the comparison and min/max paths. Done means native comparisons and aggregations match the stated Spark SQL semantics without changing only grouping-key behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, scala
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.