apache / apache/gluten

[RAS] `GroupLeafExec` does not preserve `outputPartitioning`

Open
#11,468 1 comment 0 reactions 1 assignee Claimed by @zhztheplayer View on GitHub
bug
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

`GroupLeafExec` in RAS (Ras-based Adaptive Search optimizer) always returns `UnknownPartitioning` for `outputPartitioning`, which causes incorrect behavior when `spark.sql.unionOutputPartitioning=true` (default in Spark 4.1).

### Background

Spark 4.1 introduced https://github.com/apache/spark/pull/51623 which allows `UnionExec` to preserve child partitioning. When all children have identical partitioning, Spark's optimizer trusts this information and may omit downstream Exchange operators.

However, in RAS, `GroupLeafExec` does not preserve the `outputPartitioning` from its wrapped plan, always returning `UnknownPartitioning`. This breaks the partitioning contract and can lead to incorrect query results.

### Workaround

Disable the feature by setting `spark.sql.unionOutputPartitioning=false` when using RAS.

### Proposed Solution

`GroupLeafExec` should override `outputPartitioning` to return the correct partitioning from the underlying plan.

### Related

- Spark PR: https://github.com/apache/spark/pull/51623
- Gluten PR: https://github.com/apache/incubator-gluten/pull/11455

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.