apache / apache/datafusion-comet

Evaluate extending collect_set input normalization to Spark 3.4 through 4.1

Open
#5,312 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:aggregation enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## Background

PR #5166 gates `collect_set` input normalization on Spark 4.2+, where Spark introduced the normalization contract.

On Spark 3.4 through 4.1, normalizing the input would make Comet collapse `-0.0` and `0.0` into one entry, matching Spark. The existing NaN incompatibility would remain because pre-4.2 Spark keeps distinct NaN entries while Comet deduplicates them.

## Trade-off

For array-typed children, Spark's recursive normalizer produces `ArrayTransform`, which Comet executes through the JVM codegen dispatcher. Removing the version gate would add that cost on older Spark versions too. If `spark.comet.exec.scalaUDF.codegen.enabled=false`, the aggregate safely falls back to Spark.

## Task

Evaluate removing or narrowing the `isSpark42Plus` gate so signed-zero behavior matches Spark 3.4 through 4.1 without unnecessarily routing nested arrays through JVM codegen. Add regression coverage and update the compatibility wording if the signed-zero divergence is removed.

Related: #5166 and [review discussion](https://github.com/apache/datafusion-comet/pull/5166#discussion_r3732293880).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.