Expose `preOptimizerRules` in session extensions API
- Dominant language
- Scala
- Stars
- 44k
- Forks
- 29.4k
- PR merge metrics
- No merged PRs in 30d
Description
Hi Spark community,
In our company we were planning to do some experiments on introducing table / operator stats from systems outside Spark. I didn't find a good injection point to do this in Spark public API.
I found PR https://github.com/apache/spark/pull/25015 could be useful but it didn't end up merging.
Another approach for us is to inject a logical optimizer rule to set stats for Spark logical plan. I found [this API](https://github.com/apache/spark/blob/dc9f559660937098ceeded113c2318c5c14ba73f/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala#L113-L117) helpful but it's not exposed.
Can we expose [the API](https://github.com/apache/spark/blob/dc9f559660937098ceeded113c2318c5c14ba73f/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala#L113-L117) or [both of the 2 APIs](https://github.com/apache/spark/blob/dc9f559660937098ceeded113c2318c5c14ba73f/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala#L113-L126) to the extensions API? Or any suggestion is welcomed. Thanks!
Contributor guide
Research direction
Start with sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala, especially the preOptimizerRules and adjacent APIs linked in the issue, then trace how session extensions are exposed. Done means the selected optimizer-rule API is available through the public session extensions API, with the scope of the exposed APIs resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, sql
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100