apache / apache/datafusion-comet
Nested array comparison does not match Spark for signed zero
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## Describe the bug
Nested floating-point elements use total-order comparison in Comet, where `-0.0` and `0.0` are distinct. Spark uses `ordering.equiv` for nested elements, which checks numeric equality first and therefore treats both signed zeros as equal.
This affects:
- `arrays_overlap` when the array elements are themselves arrays or structs containing floats.
- The nested fallback in `array_position`.
The flat `arrays_overlap` path is intentionally different: Spark uses `Double.equals` there and distinguishes signed zeros.
## Steps to reproduce
```sql
CREATE TABLE t(
a ARRAY>,
b ARRAY>,
v ARRAY
) USING parquet;
INSERT INTO t VALUES (
array(array(0.0D)),
array(array(-0.0D)),
array(-0.0D)
);
SELECT arrays_overlap(a, b), array_position(a, v) FROM t;
```
Spark returns `true` and `1`; Comet returns `false` and `0`.
## Expected behavior
Nested `arrays_overlap` and `array_position` should treat `-0.0` as equal to `0.0`, matching Spark, while continuing to treat NaN as equal to NaN.
## Additional context
Tracked separately from #5176 because the mismatch predates that refactor. Arrow's `make_comparator` uses total ordering; a potential fix is to normalize negative zero in nested float leaves before constructing the comparator.
Contributor guide
Research direction
Start with the nested arrays_overlap and array_position paths using the SQL reproduction in the issue, then inspect Arrow's make_comparator behavior for nested floating-point leaves. Done means nested comparisons match Spark for -0.0 and 0.0 while still treating NaN as equal, without changing the documented flat arrays_overlap behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, scala
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100