apache / apache/datafusion-comet

Enable SPARK-57298 collect_set tests when Spark 4.2 SQL diff lands

Open
#5,209 1 comment 1 reaction 1 assignee Claimed by @peterxcli View on GitHub
area:aggregation enhancement spark sql tests
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## Background

PR #5166 fixes #4966 and adds Comet SQL-file coverage for Spark 4.2 `collect_set` normalization.

The repository currently has no `dev/diffs/4.2*.diff` and no Spark 4.2 Spark SQL Tests job, so the corresponding upstream Spark tests cannot yet be enabled in the Comet diff.

## Task

Once a Spark 4.2 diff and CI job are added, enable these `DataFrameAggregateSuite` tests:

- `SPARK-57298: collect_set normalizes NaN and -0.0 for floating-point types`
- `SPARK-57298: collect_set normalizes NaN and -0.0 nested in complex types`

Verify both tests pass with native `collect_set`.

Related: #4966, #5166, and [review 4830920688](https://github.com/apache/datafusion-comet/pull/5166#pullrequestreview-4830920688).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.