apache / apache/datafusion-comet

[Bug] array_min and array_max differ from Spark on signed-zero ties

Open
#5,401 0 comments 0 reactions 0 assignees View on GitHub
area:expressions array expressions bug correctness priority:critical
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## Describe the bug

`array_min` and `array_max` do not preserve Spark's choice of the first element when `+0.0` and `-0.0` compare equal. The corrected fixtures in #5393 expose this for `array_min`. The reverse zero order also exposes it for `array_max`.

At upstream commit `91f9fbed24ee28f99b23ed559e80e818c9696044`, both expressions are classified as compatible without checking `spark.comet.exec.strictFloatingPoint`. They therefore use the differing native implementation even when strict floating-point behavior is requested.

## Reproduction

Use a Parquet column so the extrema are evaluated at runtime:

```sql
SET spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ConstantFolding;
SET spark.comet.exec.strictFloatingPoint=true;

CREATE TABLE comet_extrema_zero_repro (id INT, arr ARRAY) USING parquet;
INSERT INTO comet_extrema_zero_repro VALUES
(1, array(double('0.0'), double('-0.0'))),
(2, array(double('-0.0'), double('0.0')));

SELECT id, array_min(arr), array_max(arr)
FROM comet_extrema_zero_repro ORDER BY id;
```

The same issue occurs for `ARRAY`.

| Input | Spark min | Native min | Spark max | Native max |
| --- | --- | --- | --- | --- |
| `[+0.0, -0.0]` | `+0.0` | `-0.0` | `+0.0` | `+0.0` |
| `[-0.0, +0.0]` | `-0.0` | `-0.0` | `-0.0` | `+0.0` |

Spark 3.5.8 and 4.0.1 preserve the first zero. A direct probe of the pinned DataFusion 54.1.0 / Arrow 58.4.0 UDFs confirms the native results for both floating-point widths and on both sides of the array-length-32 kernel boundary.

## Expected behavior

Strict floating-point mode should use Spark-compatible evaluation. The native implementation should eventually preserve Spark's first-equal-element behavior as well. A narrow first fix can route strict-mode extrema through the existing Spark-codegen fallback, while keeping this issue open for native parity.

Relevant code: `CometArrayMin` and `CometArrayMax` in `spark/src/main/scala/org/apache/comet/serde/arrays.scala`. This is a follow-up to #5271 and #5393, not a regression introduced by those fixture changes.

Contributor guide

Open the contributing guide

Research direction

Start in spark/src/main/scala/org/apache/comet/serde/arrays.scala, focusing on CometArrayMin and CometArrayMax, and run the supplied Parquet SQL reproduction with strict floating-point mode enabled. Check how compatibility is classified and how the existing Spark-codegen fallback is selected. Done means strict mode preserves Spark’s first-equal-element behavior for signed-zero FLOAT and DOUBLE arrays, while native parity remains covered by the issue’s stated scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.