[CORE] Explore the need for injecting Gluten's query planner with `injectQueryStagePrepRule`
Open
enhancement
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
So far Gluten's query planner is injected to Spark via columnar rule extension. In the context of AQE, this practice may cause us to lose chance of generating a global-optimal query plan. For example, we may want to decide whether we should use GPU shuffle write for last stage when analyzing the current stage's query plan.
To address this, we'd explore the feasibility of injecting Gluten's query planner via `injectQueryStagePrepRule` as well.
Contributor guide
Assessment
This issue has not been assessed yet.