Native float ordering diverges from Spark for NaN and signed zero
- Dominant language
- Rust
- Stars
- 1.8k
- Forks
- 241
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 22
Description
**Describe the bug**
Native floating-point comparisons and min/max aggregations diverge from Spark SQL semantics for NaN and signed zero (-0.0 vs 0.0).
**To Reproduce**
Run the following DataFrame queries:
```scala
val maxDouble = Seq(Double.NaN, Double.PositiveInfinity, Double.MaxValue)
.map(Tuple1.apply).toDF("a").selectExpr("max(a)").first()
assert(java.lang.Double.isNaN(maxDouble.getDouble(0)))
val comparison = Seq((0.0, -0.0)).toDF("pos", "neg")
.select($"pos" > $"neg").first()
assert(comparison.getBoolean(0) == false)
```
Actual Auron results:
- Native `max(a)` returns a non-NaN value (`Double.PositiveInfinity`) instead of `NaN`.
- Native `0.0 > -0.0` evaluates to `true`.
Reproduced on Spark 3.1 and Spark 3.5.
**Expected behavior**
Spark SQL floating-point ordering semantics:
- `NaN` is equal to `NaN` and greater than any non-NaN value.
- `-0.0` is equal to `0.0`.
`max(a)` should return `NaN` when `NaN` is present, and `0.0 > -0.0` should return `false`.
**Screenshots**
N/A
**Additional context**
- Affected upstream commit: master at `9f90eacb`
- Upstream source file and line references:
- `spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeConverters.scala:576-581`
- `native-engine/datafusion-ext-plans/src/agg/maxmin.rs:110-124, 207-220`
- Distinction from apache/auron#1015 and apache/auron#1016: those changes added `NormalizeNaNAndZero` for grouping keys, but direct comparisons and native min/max aggregations still use native float comparisons directly.
Contributor guide
Research direction
Start by reproducing the NaN and signed-zero cases from the issue in Spark and Auron. Read spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeConverters.scala:576-581 and native-engine/datafusion-ext-plans/src/agg/maxmin.rs:110-124, 207-220, then trace the comparison and min/max paths. Done means native comparisons and aggregations match the stated Spark SQL semantics without changing only grouping-key behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, scala
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100