[VL] Gluten plugin graceful shutdown when Spark session is closed
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
Spark session can be shut down and renewed per Spark's design. When a new session is created, some session-wise configurations (e.g. Spark local dir) may change. Gluten now only do static initialization once in the same process without checking configuration changes.
We should implement a new mechanism to clean up Gluten's global states as well as vanilla Spark's doing.
Contributor guide
Research direction
Start by tracing Gluten's static initialization and how it relates to Spark session creation and closure. Compare the lifecycle with vanilla Spark's cleanup behavior and identify the session-wise configuration changes that must be handled. Done means Gluten global state is cleaned up and correctly reinitialized when a Spark session is closed and renewed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100