apache / apache/datafusion-comet
Spark 4.2: Comet native collect_set does not normalize NaN / -0.0 (SPARK-57298)
Open
area:aggregation
bug
priority:critical
spark 4.2
spark sql tests
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
Spark 4.2 (SPARK-57298) makes `collect_set` normalize `NaN` and `-0.0` for floating-point types. Comet's native `collect_set` aggregate does not perform this normalization, so results differ from Spark.
Failing Spark SQL test (currently marked `IgnoreComet` in `dev/diffs/4.2.0.diff`):
- `DataFrameAggregateSuite`: "SPARK-57298: collect_set normalizes NaN and -0.0 for floating-point types"
Note: the array-expression normalization path (`array_distinct` / `array_except`) is already handled; this issue tracks the aggregate `collect_set` path.
Contributor guide
Assessment
This issue has not been assessed yet.