apache / apache/spark

Expose `preOptimizerRules` in session extensions API

Open
#54,687 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
44k
Forks
29.4k
PR merge metrics
No merged PRs in 30d

Description

Hi Spark community,

In our company we were planning to do some experiments on introducing table / operator stats from systems outside Spark. I didn't find a good injection point to do this in Spark public API.

I found PR https://github.com/apache/spark/pull/25015 could be useful but it didn't end up merging.

Another approach for us is to inject a logical optimizer rule to set stats for Spark logical plan. I found [this API](https://github.com/apache/spark/blob/dc9f559660937098ceeded113c2318c5c14ba73f/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala#L113-L117) helpful but it's not exposed.

Can we expose [the API](https://github.com/apache/spark/blob/dc9f559660937098ceeded113c2318c5c14ba73f/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala#L113-L117) or [both of the 2 APIs](https://github.com/apache/spark/blob/dc9f559660937098ceeded113c2318c5c14ba73f/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala#L113-L126) to the extensions API? Or any suggestion is welcomed. Thanks!

Contributor guide

Open the contributing guide

Research direction

Start with sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala, especially the preOptimizerRules and adjacent APIs linked in the issue, then trace how session extensions are exposed. Done means the selected optimizer-rule API is available through the public session extensions API, with the scope of the exposed APIs resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, sql
Domain
backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.