apache / apache/datafusion-comet

Nested array comparison does not match Spark for signed zero

Open
#5,191 1 comment 0 reactions 0 assignees View on GitHub
correctness good first issue priority:high
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## Describe the bug

Nested floating-point elements use total-order comparison in Comet, where `-0.0` and `0.0` are distinct. Spark uses `ordering.equiv` for nested elements, which checks numeric equality first and therefore treats both signed zeros as equal.

This affects:

- `arrays_overlap` when the array elements are themselves arrays or structs containing floats.
- The nested fallback in `array_position`.

The flat `arrays_overlap` path is intentionally different: Spark uses `Double.equals` there and distinguishes signed zeros.

## Steps to reproduce

```sql
CREATE TABLE t(
a ARRAY>,
b ARRAY>,
v ARRAY
) USING parquet;

INSERT INTO t VALUES (
array(array(0.0D)),
array(array(-0.0D)),
array(-0.0D)
);

SELECT arrays_overlap(a, b), array_position(a, v) FROM t;
```

Spark returns `true` and `1`; Comet returns `false` and `0`.

## Expected behavior

Nested `arrays_overlap` and `array_position` should treat `-0.0` as equal to `0.0`, matching Spark, while continuing to treat NaN as equal to NaN.

## Additional context

Tracked separately from #5176 because the mismatch predates that refactor. Arrow's `make_comparator` uses total ordering; a potential fix is to normalize negative zero in nested float leaves before constructing the comparator.

Contributor guide

Open the contributing guide

Research direction

Start with the nested arrays_overlap and array_position paths using the SQL reproduction in the issue, then inspect Arrow's make_comparator behavior for nested floating-point leaves. Done means nested comparisons match Spark for -0.0 and 0.0 while still treating NaN as equal, without changing the documented flat arrays_overlap behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, scala
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.