apache / apache/gluten

[VL] Gluten plugin graceful shutdown when Spark session is closed

Open
#6,862 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Spark session can be shut down and renewed per Spark's design. When a new session is created, some session-wise configurations (e.g. Spark local dir) may change. Gluten now only do static initialization once in the same process without checking configuration changes.

We should implement a new mechanism to clean up Gluten's global states as well as vanilla Spark's doing.

Contributor guide

Open the contributing guide

Research direction

Start by tracing Gluten's static initialization and how it relates to Spark session creation and closure. Compare the lifecycle with vanilla Spark's cleanup behavior and identify the session-wise configuration changes that must be handled. Done means Gluten global state is cleaned up and correctly reinitialized when a Spark session is closed and renewed.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.