apache / apache/datafusion-comet
Evaluate extending collect_set input normalization to Spark 3.4 through 4.1
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## Background
PR #5166 gates `collect_set` input normalization on Spark 4.2+, where Spark introduced the normalization contract.
On Spark 3.4 through 4.1, normalizing the input would make Comet collapse `-0.0` and `0.0` into one entry, matching Spark. The existing NaN incompatibility would remain because pre-4.2 Spark keeps distinct NaN entries while Comet deduplicates them.
## Trade-off
For array-typed children, Spark's recursive normalizer produces `ArrayTransform`, which Comet executes through the JVM codegen dispatcher. Removing the version gate would add that cost on older Spark versions too. If `spark.comet.exec.scalaUDF.codegen.enabled=false`, the aggregate safely falls back to Spark.
## Task
Evaluate removing or narrowing the `isSpark42Plus` gate so signed-zero behavior matches Spark 3.4 through 4.1 without unnecessarily routing nested arrays through JVM codegen. Add regression coverage and update the compatibility wording if the signed-zero divergence is removed.
Related: #5166 and [review discussion](https://github.com/apache/datafusion-comet/pull/5166#discussion_r3732293880).
Contributor guide
Assessment
This issue has not been assessed yet.