apache / apache/datafusion-comet

Spark 4.2: UnionCodegenSuite partitioning-aware test inspects UnionExec that Comet replaces

Open
#4,965 0 comments 0 reactions 0 assignees View on GitHub
bug priority:low spark 4.2 spark sql tests
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

Spark 4.2's `UnionCodegenSuite` (SPARK-56482) test "partitioning-aware union falls back to non-codegen" inspects Spark's `UnionExec` whole-stage-codegen internals via `executedPlan.collectFirst { case u: UnionExec => u }.get`.

Comet replaces `UnionExec` with `CometUnionExec`, so `collectFirst` returns `None` and the test fails with `NoSuchElementException: None.get`.

Currently marked `IgnoreComet` in `dev/diffs/4.2.0.diff` (`WholeStageCodegenSuite` is already `IgnoreCometSuite`).

Contributor guide

Open the contributing guide

Research direction

Start with the Spark 4.2 UnionCodegenSuite test named "partitioning-aware union falls back to non-codegen" and the corresponding entry in dev/diffs/4.2.0.diff. Check how Comet handles UnionExec replacement and compare the existing WholeStageCodegenSuite IgnoreComet treatment. Done means the test no longer fails with None.get when Comet is enabled, with the intended compatibility behavior recorded in the diff.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.