apache / apache/datafusion-comet
Spark 4.2: UnionCodegenSuite partitioning-aware test inspects UnionExec that Comet replaces
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
Spark 4.2's `UnionCodegenSuite` (SPARK-56482) test "partitioning-aware union falls back to non-codegen" inspects Spark's `UnionExec` whole-stage-codegen internals via `executedPlan.collectFirst { case u: UnionExec => u }.get`.
Comet replaces `UnionExec` with `CometUnionExec`, so `collectFirst` returns `None` and the test fails with `NoSuchElementException: None.get`.
Currently marked `IgnoreComet` in `dev/diffs/4.2.0.diff` (`WholeStageCodegenSuite` is already `IgnoreCometSuite`).
Contributor guide
Research direction
Start with the Spark 4.2 UnionCodegenSuite test named "partitioning-aware union falls back to non-codegen" and the corresponding entry in dev/diffs/4.2.0.diff. Check how Comet handles UnionExec replacement and compare the existing WholeStageCodegenSuite IgnoreComet treatment. Done means the test no longer fails with None.get when Comet is enabled, with the intended compatibility behavior recorded in the diff.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 48/100