apache / apache/datafusion-comet

Spark 4.2: `OneRowRelation` in Union branches forces Union and downstream aggregates off Comet (TPC-DS q77a)

Open
#4,949 0 comments 0 reactions 0 assignees View on GitHub
area:scan bug performance priority:medium spark 4.2
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

https://github.com/apache/datafusion-comet/issues/4142

## Describe the bug

On Spark 4.2.0, TPC-DS `q77a` loses Comet acceleration for a large part of its plan. This showed up while regenerating the TPC-DS plan-stability golden files for the released Spark 4.2.0 (previously the `spark-4.2` profile targeted `4.2.0-preview4`).

Comparing the approved plan for `q77a` against the fallback baseline (`approved-plans-v2_7/q77a`):

| | `approved-plans-v2_7` (3.4 - 4.1) | `approved-plans-v2_7-spark4_2` |
| --- | --- | --- |
| `CometUnion` | 4 | **0** |
| bare `Union` | 0 | **4** |
| non-Comet `Exchange` / `HashAggregate` | 0 | **12** |
| `Scan OneRowRelation` | 0 | **3** |

## Root cause

Spark 4.2 plans a `OneRowRelation` into the `Union` branches of this query (it is absent from the plan on 3.4 - 4.1). Comet does not support it:

```
+- Scan OneRowRelation [COMET: Scan OneRowRelation is not supported]
```

That single unsupported leaf cascades:

1. The `Union` can no longer be converted, so all 4 `CometUnion` become bare `Union`.
2. Its Comet children get capped with `CometNativeColumnarToRow`.
3. The `Final` / `PartialMerge` aggregates above the `Union` then have no Comet partial aggregate producing their intermediate buffers, so the guard in `CometBaseAggregate.doConvert` (`spark/src/main/scala/org/apache/spark/sql/comet/operators.scala:1639`) rejects them:

```
HashAggregate [COMET: Comet aggregate that merges intermediate buffers requires a Comet child
aggregate when the intermediate buffer formats are incompatible with Spark.
Incompatible aggregate function(s): sum]
```

## Expected behavior

`q77a` should keep the same degree of Comet acceleration on Spark 4.2 that it has on 3.4 - 4.1 (fully native `CometUnion` + `CometHashAggregate`).

## Additional context

`OneRowRelation` is already listed in the default value of `spark.comet.sparkToColumnar.supportedOperatorList` (`spark/src/main/scala/org/apache/comet/CometConf.scala:801`), so routing it through the Spark-to-Arrow path may be most of the fix. The `adding_a_new_spark_version.md` contributor guide also calls out "a new `OneRowRelation` planning path" as a known Spark 4.x behavioral difference.

The regenerated `approved-plans-v2_7-spark4_2/q77a` golden bakes in the degraded plan so that CI is green for the Spark 4.2.0 bring-up. That golden should be removed (letting `q77a` fall back to the shared baseline plan) as part of fixing this.

Contributor guide

Open the contributing guide

Research direction

Start by reading spark/src/main/scala/org/apache/comet/CometConf.scala around the supported operator list and spark/src/main/scala/org/apache/spark/sql/comet/operators.scala around CometBaseAggregate.doConvert. Compare the Spark 4.2 q77a plan with approved-plans-v2_7/q77a and consult adding_a_new_spark_version.md for the OneRowRelation planning difference. Done means q77a retains CometUnion and CometHashAggregate acceleration, and the degraded approved-plans-v2_7-spark4_2/q77a golden is removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.