apache / apache/gluten

[CORE] Explore the need for injecting Gluten's query planner with `injectQueryStagePrepRule`

Open
#11,053 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

So far Gluten's query planner is injected to Spark via columnar rule extension. In the context of AQE, this practice may cause us to lose chance of generating a global-optimal query plan. For example, we may want to decide whether we should use GPU shuffle write for last stage when analyzing the current stage's query plan.

To address this, we'd explore the feasibility of injecting Gluten's query planner via `injectQueryStagePrepRule` as well.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.