apache / apache/gluten

Incompatible issues of cross stage aggregate operators

Open
#3,391 4 comments 0 reactions 0 assignees View on GitHub
enhancement on track
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Spark sort agg will be replaced by gluten hash agg if it is transformable. But it is possible that partial agg is offloaded, but final agg cannot due to agg's post project is not supported. As partial hash agg is not compatible with final sort agg, there will have runtime issues. ObjectHash agg should also have such similar issue.

One internally proposed solution is, moving unsupported post project to an independent project operator. Thus, we can make final agg transformed and just make post project fall back.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.