feat(spark): support connect_timeout when connecting to an existing Spark Connect server via base_url
- Dominant language
- Python
- Stars
- 148
- Forks
- 262
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 1
Description
What you would like to be added?
SparkClient.connect() already supports timeout and connect_timeout when creating a new Spark Connect session through create_and_connect().
However, when base_url is provided, the method directly calls:
SparkSession.builder.remote(base_url).getOrCreate()
without applying any timeout handling.
We should extend the existing connect_timeout behavior to the base_url connection path so users can configure a maximum time to wait when connecting to an existing Spark Connect server.
Example:
from kubeflow.spark import SparkClient
client = SparkClient(...)
spark = client.connect(
base_url="sc://spark-connect:15002",
connect_timeout=30,
)
If the connection cannot be established within the configured timeout, the SDK should raise a clear TimeoutError, similar to the existing session creation flow.
Why is this needed?
The session creation flow already enforces connect_timeout, but the direct base_url connection path does not.
As a result, applications connecting to an existing Spark Connect server may block indefinitely if the endpoint is unreachable or unresponsive.
Supporting connect_timeout consistently in both connection paths would provide a uniform API and allow applications to fail fast with a clear TimeoutError.
Love this feature?
Give it a 👍 We prioritize the features with most 👍
Contributor guide
Research direction
Start with SparkClient.connect() and compare the base_url path using SparkSession.builder.remote(base_url).getOrCreate() with the existing create_and_connect() timeout handling. Make connect_timeout apply to an existing Spark Connect server and verify that an unreachable endpoint raises a clear TimeoutError rather than blocking indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100