apache / apache/datafusion-comet
[Bug] array_min and array_max differ from Spark on signed-zero ties
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## Describe the bug
`array_min` and `array_max` do not preserve Spark's choice of the first element when `+0.0` and `-0.0` compare equal. The corrected fixtures in #5393 expose this for `array_min`. The reverse zero order also exposes it for `array_max`.
At upstream commit `91f9fbed24ee28f99b23ed559e80e818c9696044`, both expressions are classified as compatible without checking `spark.comet.exec.strictFloatingPoint`. They therefore use the differing native implementation even when strict floating-point behavior is requested.
## Reproduction
Use a Parquet column so the extrema are evaluated at runtime:
```sql
SET spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ConstantFolding;
SET spark.comet.exec.strictFloatingPoint=true;
CREATE TABLE comet_extrema_zero_repro (id INT, arr ARRAY) USING parquet;
INSERT INTO comet_extrema_zero_repro VALUES
(1, array(double('0.0'), double('-0.0'))),
(2, array(double('-0.0'), double('0.0')));
SELECT id, array_min(arr), array_max(arr)
FROM comet_extrema_zero_repro ORDER BY id;
```
The same issue occurs for `ARRAY`.
| Input | Spark min | Native min | Spark max | Native max |
| --- | --- | --- | --- | --- |
| `[+0.0, -0.0]` | `+0.0` | `-0.0` | `+0.0` | `+0.0` |
| `[-0.0, +0.0]` | `-0.0` | `-0.0` | `-0.0` | `+0.0` |
Spark 3.5.8 and 4.0.1 preserve the first zero. A direct probe of the pinned DataFusion 54.1.0 / Arrow 58.4.0 UDFs confirms the native results for both floating-point widths and on both sides of the array-length-32 kernel boundary.
## Expected behavior
Strict floating-point mode should use Spark-compatible evaluation. The native implementation should eventually preserve Spark's first-equal-element behavior as well. A narrow first fix can route strict-mode extrema through the existing Spark-codegen fallback, while keeping this issue open for native parity.
Relevant code: `CometArrayMin` and `CometArrayMax` in `spark/src/main/scala/org/apache/comet/serde/arrays.scala`. This is a follow-up to #5271 and #5393, not a regression introduced by those fixture changes.
Contributor guide
Research direction
Start in spark/src/main/scala/org/apache/comet/serde/arrays.scala, focusing on CometArrayMin and CometArrayMax, and run the supplied Parquet SQL reproduction with strict floating-point mode enabled. Check how compatibility is classified and how the existing Spark-codegen fallback is selected. Done means strict mode preserves Spark’s first-equal-element behavior for signed-zero FLOAT and DOUBLE arrays, while native parity remains covered by the issue’s stated scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100