kubeflow / kubeflow/sdk

SparkConnect Idle clean-up

Open
#800 1 comment 0 reactions 0 assignees View on GitHub
kind/feature needs-triage
Dominant language
Python
Stars
148
Forks
262
Avg merge
1d 2h
Merged PRs (30d)
1

Description

### What you would like to be added?

Add automatic cleanup for Spark Connect sessions after a configurable idle period so that temporary Spark Connect resources do not remain running indefinitely.

### Why is this needed?

The current `kubeflow.spark` SDK supports creating and connecting to Spark Connect sessions, but it does not appear to provide any built-in idle timeout or automatic deletion behavior for those sessions.

This means a Spark Connect resource can remain alive in Kubernetes even after the client is no longer actively using it, leading to:

- wasted cluster resources
- orphaned Spark Connect sessions
- operational drift when many users create ad hoc sessions

### Love this feature?

Give it a 👍 We prioritize the features with most 👍

Contributor guide

Open the contributing guide

Research direction

Start by locating the kubeflow.spark SDK entry points for creating, connecting to, and deleting Spark Connect sessions. Review how session configuration is exposed and determine the expected idle-timeout behavior, cleanup trigger, and handling of active sessions. Done means configurable idle sessions are cleaned up without deleting sessions still in use, with tests covering the lifecycle.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python, spark
Domain
cloud, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.