apache / apache/datafusion-comet

Signed-zero array test fixtures claim the literal case agrees with Spark when it does not

Open Beginner friendly
#5,702 1 comment 0 reactions 0 assignees View on GitHub
area:expressions priority:low requires-triage test
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### Describe the bug

The signed-zero fixtures in the array SQL tests carry comments that contradict the tests around them, and would mislead anyone deciding whether to re-enable them.

`spark/src/test/resources/sql-tests/expressions/array/array_distinct.sql:139`:

```
-- negative zero (literal). Spark's NormalizeFloatingNumbers rewrites -0.0 to 0.0 at
-- analysis time, so both Spark and Comet collapse it and agree here.
query ignore(https://issues.apache.org/jira/browse/SPARK-54918)
SELECT array_distinct(array(0.0, double('-0.0'), 1.0))
```

The comment says the two agree, but the query immediately below it is skipped precisely because they do not. `NormalizeFloatingNumbers` only rewrites grouping keys, join keys, window partition specs and `Distinct`; the optimized plan for a plain `SELECT` keeps the `-0.0` literal intact.

The same "NormalizeFloatingNumbers only rewrites literals, not parquet columns" framing appears in `array_except.sql:77`, `array_intersect.sql:162` and `array_union.sql:149`.

### Steps to reproduce

Read the fixtures, or run `SELECT array_distinct(array(0.0, double('-0.0'), 1.0))` and observe that the literal case diverges too.

### Expected behavior

The comments describe what actually happens: the divergence is not limited to column-sourced values, and the literal case is skipped for the same reason.

### Additional context

Depends on the behavior decision in the linked signed-zero issue; the comments should be corrected when the skips are resolved. Found while reviewing #5262.

Contributor guide

Open the contributing guide

Research direction

Read the comments and nearby queries in spark/src/test/resources/sql-tests/expressions/array/array_distinct.sql:139, array_except.sql:77, array_intersect.sql:162, and array_union.sql:149. Check the referenced signed-zero behavior and update the comments so they match the skipped cases and the eventual behavior decision. Done means all four fixture comments accurately explain the literal and column cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark, sql
Domain
testing
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.