apache / apache/datafusion-comet

array_distinct and array_union diverge from Spark on -0.0 for Spark versions without SPARK-54918

Open
#5,701 2 comments 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:expressions bug correctness priority:high requires-triage
Dominant language
Scala
Stars
1.3k
Forks
375
Avg merge
2d 4h
Merged PRs (30d)
213

Description

### Describe the bug

`array_distinct` and `array_union` run natively by default (`ArrayDistinct` via a plain `CometScalarFunction`, `ArrayUnion` via `CometArrayUnion`; neither overrides `getSupportLevel`). DataFusion collapses `-0.0` and `0.0`, so on a Spark version that does not normalize signed zeros in array functions, Comet silently returns a different answer:

```
SELECT array_distinct(array(0.0, double('-0.0'), 1.0))
Spark: [0.0, -0.0, 1.0]
Comet: [0.0, 1.0]
```

The plan is `CometProject` with no fallback. `NormalizeFloatingNumbers` only rewrites grouping keys, join keys, window partition specs and `Distinct`, so a plain `SELECT` diverges for literals as well as column-sourced values.

[SPARK-54918](https://issues.apache.org/jira/browse/SPARK-54918) makes Spark normalize signed zeros in `array_distinct`, `array_union`, `array_intersect`, `array_except` and `arrays_overlap`, which makes `[0.0, 1.0]` the correct answer. Its fix versions are **4.2.0, 4.1.4 and 4.0.5**. Comet also supports 3.4 and 3.5, which will never get it, and any 4.0.x before 4.0.5 or 4.1.x before 4.1.4.

`array_intersect` and `array_except` already report `Incompatible`, so they only reach the native path under `allowIncompatible`. `array_distinct` and `array_union` do not.

### Steps to reproduce

Run the query above on Spark 3.5, or on 4.1.3 (the version the build currently pins).

### Expected behavior

On a Spark version without SPARK-54918, either match Spark or fall back. On a version with it, keep the native path.

### Additional context

The split is at the **patch** level, so a compile-time shim keyed on `shims.minorVerSrc` is not sufficient: the build pins 4.0.4 and 4.1.3, but Comet runs against any patch release of those lines. It needs a runtime check.

Note that `CometSparkSessionExtensions.isSparkNNPlus` compares `org.apache.spark.SPARK_VERSION` as a **string**, so `SPARK_VERSION >= "4.0.5"` would be wrong once 4.0.10 ships. A numeric comparison helper is needed.

Suggested shape: a `getSupportLevel` override on the `ArrayDistinct` and `ArrayUnion` serdes returning `Incompatible` when the element type contains `FloatType`/`DoubleType` and the running Spark version predates the fix, in the shape `CometArrayExcept` already uses for its unsupported element types. That keeps the fast path for every other element type and every fixed Spark version.

The signed-zero cases in `array_distinct.sql`, `array_union.sql`, `array_except.sql` and `array_intersect.sql` are currently `query ignore(...)`; they should be restored once behavior is version-correct.

Found while reviewing #5262.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.