kubeflow / kubeflow/sdk

feat(spark): support connect_timeout when connecting to an existing Spark Connect server via base_url

Open
#650 2 comments 0 reactions 0 assignees View on GitHub
area/spark kind/feature
Dominant language
Python
Stars
148
Forks
262
Avg merge
1d 2h
Merged PRs (30d)
1

Description

What you would like to be added?

SparkClient.connect() already supports timeout and connect_timeout when creating a new Spark Connect session through create_and_connect().
However, when base_url is provided, the method directly calls:

SparkSession.builder.remote(base_url).getOrCreate()

without applying any timeout handling.
We should extend the existing connect_timeout behavior to the base_url connection path so users can configure a maximum time to wait when connecting to an existing Spark Connect server.

Example:

from kubeflow.spark import SparkClient

client = SparkClient(...)

spark = client.connect(
base_url="sc://spark-connect:15002",
connect_timeout=30,
)

If the connection cannot be established within the configured timeout, the SDK should raise a clear TimeoutError, similar to the existing session creation flow.

Why is this needed?

The session creation flow already enforces connect_timeout, but the direct base_url connection path does not.

As a result, applications connecting to an existing Spark Connect server may block indefinitely if the endpoint is unreachable or unresponsive.

Supporting connect_timeout consistently in both connection paths would provide a uniform API and allow applications to fail fast with a clear TimeoutError.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

Contributor guide

Open the contributing guide

Research direction

Start with SparkClient.connect() and compare the base_url path using SparkSession.builder.remote(base_url).getOrCreate() with the existing create_and_connect() timeout handling. Make connect_timeout apply to an existing Spark Connect server and verify that an unreachable endpoint raises a clear TimeoutError rather than blocking indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, spark
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.