Fix in-cluster Spark Connect URL: use .svc instead of .svc.cluster.local
- Dominant language
- Python
- Stars
- 148
- Forks
- 262
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 1
Description
### What happened?
When `SparkClient` runs **in-cluster** (for example from a notebook pod or an E2E job inside the cluster), `KubernetesBackend` builds the Spark Connect URL via `build_service_url()` in [kubeflow/spark/backends/kubernetes/utils.py](https://github.com/kubeflow/sdk/blob/0.4.0/kubeflow/spark/backends/kubernetes/utils.py#L84):
```python
return f"sc://{service}.{info.namespace}.svc.cluster.local:{constants.SPARK_CONNECT_PORT}"
```
That produces URLs like:
`sc://my-session-svc.spark.svc.cluster.local:15002`
This does not match the documented/expected Kubernetes cluster DNS short form (`..svc`) and diverges from KEP-107 examples (`sc://spark-cluster.spark-system.svc:15002`).
### What did you expect to happen?
In-cluster URLs should use the standard cluster DNS suffix:
```python
return f"sc://{service}.{info.namespace}.svc:{constants.SPARK_CONNECT_PORT}"
```
Example:
`sc://my-session-svc.spark.svc:15002`
**Expected behavior**
- In-cluster `SparkClient.connect()` (create mode, no `base_url`) resolves the session service via `-svc..svc:`.
- `build_service_url()` and unit tests align with this format.
- Comments in e2e/hack docs that still mention `svc.cluster.local` are updated for consistency.
**Suggested scope**
- Change `build_service_url()` in [kubeflow/spark/backends/kubernetes/utils.py](https://github.com/kubeflow/sdk/blob/0.4.0/kubeflow/spark/backends/kubernetes/utils.py#L84):
- Update [kubeflow/spark/backends/kubernetes/utils_test.py](https://github.com/kubeflow/sdk/blob/0.4.0/kubeflow/spark/backends/kubernetes/utils_test.py) (and any backend tests asserting `svc.cluster.local`).
- Refresh e2e/doc comments if they reference the old hostname.
### Environment
Kubernetes version:
```bash
$ kubectl version
Client Version: v1.35.3
Kustomize Version: v5.7.1
Server Version: v1.33.0
```
Kubeflow Trainer version:
```bash
$ kubectl get pods -n kubeflow -l app.kubernetes.io/name=trainer -o jsonpath="{.items[*].spec.containers[*].image}"
```
Kubeflow Python SDK version:
```bash
$ pip show kubeflow
0.4.0
```
### Impacted by this bug?
Give it a 👍 We prioritize the issues with most 👍
Contributor guide
Research direction
Start with build_service_url() in kubeflow/spark/backends/kubernetes/utils.py and run kubeflow/spark/backends/kubernetes/utils_test.py. Search backend tests and e2e or hack comments for svc.cluster.local references. Done means in-cluster URLs and their tests use the documented .svc form consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100