Incompatible issues of cross stage aggregate operators
Open
enhancement
on track
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
Spark sort agg will be replaced by gluten hash agg if it is transformable. But it is possible that partial agg is offloaded, but final agg cannot due to agg's post project is not supported. As partial hash agg is not compatible with final sort agg, there will have runtime issues. ObjectHash agg should also have such similar issue.
One internally proposed solution is, moving unsupported post project to an independent project operator. Thus, we can make final agg transformed and just make post project fall back.
Contributor guide
Assessment
This issue has not been assessed yet.