SparkConnect Idle clean-up
- Dominant language
- Python
- Stars
- 148
- Forks
- 262
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 1
Description
### What you would like to be added?
Add automatic cleanup for Spark Connect sessions after a configurable idle period so that temporary Spark Connect resources do not remain running indefinitely.
### Why is this needed?
The current `kubeflow.spark` SDK supports creating and connecting to Spark Connect sessions, but it does not appear to provide any built-in idle timeout or automatic deletion behavior for those sessions.
This means a Spark Connect resource can remain alive in Kubernetes even after the client is no longer actively using it, leading to:
- wasted cluster resources
- orphaned Spark Connect sessions
- operational drift when many users create ad hoc sessions
### Love this feature?
Give it a 👍 We prioritize the features with most 👍
Contributor guide
Research direction
Start by locating the kubeflow.spark SDK entry points for creating, connecting to, and deleting Spark Connect sessions. Review how session configuration is exposed and determine the expected idle-timeout behavior, cleanup trigger, and handling of active sessions. Done means configurable idle sessions are cleaned up without deleting sessions still in use, with tests covering the lifecycle.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python, spark
- Domain
- cloud, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100