[RAS] `GroupLeafExec` does not preserve `outputPartitioning`
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
`GroupLeafExec` in RAS (Ras-based Adaptive Search optimizer) always returns `UnknownPartitioning` for `outputPartitioning`, which causes incorrect behavior when `spark.sql.unionOutputPartitioning=true` (default in Spark 4.1).
### Background
Spark 4.1 introduced https://github.com/apache/spark/pull/51623 which allows `UnionExec` to preserve child partitioning. When all children have identical partitioning, Spark's optimizer trusts this information and may omit downstream Exchange operators.
However, in RAS, `GroupLeafExec` does not preserve the `outputPartitioning` from its wrapped plan, always returning `UnknownPartitioning`. This breaks the partitioning contract and can lead to incorrect query results.
### Workaround
Disable the feature by setting `spark.sql.unionOutputPartitioning=false` when using RAS.
### Proposed Solution
`GroupLeafExec` should override `outputPartitioning` to return the correct partitioning from the underlying plan.
### Related
- Spark PR: https://github.com/apache/spark/pull/51623
- Gluten PR: https://github.com/apache/incubator-gluten/pull/11455
Contributor guide
Assessment
This issue has not been assessed yet.