apache / apache/datafusion-comet

Avoid `CometColumnarExchange` when next query stage requires `CometColumnarToRow`

Open
#2,880 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

The golden file at `spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q1.native_datafusion/extended.txt` has this section:

```
: : : +- HashAggregate
: : : +- CometColumnarToRow
: : : +- CometColumnarExchange
: : : +- HashAggregate
```

Presumably the first `HashAggregate` shown here is not supported by Comet (fallback reasons are missing for some reason) so this query stage needs to call `CometColumnarToRow`.

There is _probably_ no benefit in using columnar shuffle in the child stage if the parent stage immediately converts back to row-based, although we should confirm that assumption.

### Describe the potential solution

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.