apache / apache/datafusion-comet
Spark 4.2: `OneRowRelation` in Union branches forces Union and downstream aggregates off Comet (TPC-DS q77a)
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
https://github.com/apache/datafusion-comet/issues/4142
## Describe the bug
On Spark 4.2.0, TPC-DS `q77a` loses Comet acceleration for a large part of its plan. This showed up while regenerating the TPC-DS plan-stability golden files for the released Spark 4.2.0 (previously the `spark-4.2` profile targeted `4.2.0-preview4`).
Comparing the approved plan for `q77a` against the fallback baseline (`approved-plans-v2_7/q77a`):
| | `approved-plans-v2_7` (3.4 - 4.1) | `approved-plans-v2_7-spark4_2` |
| --- | --- | --- |
| `CometUnion` | 4 | **0** |
| bare `Union` | 0 | **4** |
| non-Comet `Exchange` / `HashAggregate` | 0 | **12** |
| `Scan OneRowRelation` | 0 | **3** |
## Root cause
Spark 4.2 plans a `OneRowRelation` into the `Union` branches of this query (it is absent from the plan on 3.4 - 4.1). Comet does not support it:
```
+- Scan OneRowRelation [COMET: Scan OneRowRelation is not supported]
```
That single unsupported leaf cascades:
1. The `Union` can no longer be converted, so all 4 `CometUnion` become bare `Union`.
2. Its Comet children get capped with `CometNativeColumnarToRow`.
3. The `Final` / `PartialMerge` aggregates above the `Union` then have no Comet partial aggregate producing their intermediate buffers, so the guard in `CometBaseAggregate.doConvert` (`spark/src/main/scala/org/apache/spark/sql/comet/operators.scala:1639`) rejects them:
```
HashAggregate [COMET: Comet aggregate that merges intermediate buffers requires a Comet child
aggregate when the intermediate buffer formats are incompatible with Spark.
Incompatible aggregate function(s): sum]
```
## Expected behavior
`q77a` should keep the same degree of Comet acceleration on Spark 4.2 that it has on 3.4 - 4.1 (fully native `CometUnion` + `CometHashAggregate`).
## Additional context
`OneRowRelation` is already listed in the default value of `spark.comet.sparkToColumnar.supportedOperatorList` (`spark/src/main/scala/org/apache/comet/CometConf.scala:801`), so routing it through the Spark-to-Arrow path may be most of the fix. The `adding_a_new_spark_version.md` contributor guide also calls out "a new `OneRowRelation` planning path" as a known Spark 4.x behavioral difference.
The regenerated `approved-plans-v2_7-spark4_2/q77a` golden bakes in the degraded plan so that CI is green for the Spark 4.2.0 bring-up. That golden should be removed (letting `q77a` fall back to the shared baseline plan) as part of fixing this.
Contributor guide
Research direction
Start by reading spark/src/main/scala/org/apache/comet/CometConf.scala around the supported operator list and spark/src/main/scala/org/apache/spark/sql/comet/operators.scala around CometBaseAggregate.doConvert. Compare the Spark 4.2 q77a plan with approved-plans-v2_7/q77a and consult adding_a_new_spark_version.md for the OneRowRelation planning difference. Done means q77a retains CometUnion and CometHashAggregate acceleration, and the degraded approved-plans-v2_7-spark4_2/q77a golden is removed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100