kubeflow / kubeflow/sdk

ClusterRuntime un-getable

Open
#335 6 comments 0 reactions 0 assignees View on GitHub
kind/bug needs-triage
Dominant language
Python
Stars
148
Forks
262
Avg merge
1d 2h
Merged PRs (30d)
1

Description

### What happened?

It looks like with recent changes to the Kubernetes backend, ClusterRuntime are not retrievable.

When attempting to get a ClusterRuntime, it can never go past the exception caught on L199:
https://github.com/kubeflow/sdk/blob/db76de0ca8c32540b8a177a713e73f41c1bf81af/kubeflow/trainer/backends/kubernetes/backend.py#L174-L223

The change seems to come from https://github.com/kubeflow/sdk/pull/307. Based on skimming that PR, it looks like it probably worked originally with the `pass` but that was pulling out due to a suggestion from copilot. It doesn't need to raise the error there as that requires something above it to catch it. It should probably just print/log it instead of catching and raising again.

Disclaimer: I am not familiar with this codebase nor have I had time to fully dig into it, so maybe I missed up something in my configuration. Either way, it warrants a deeper look.

### What did you expect to happen?

`get_runtime` should first try namespaced runtimes and then fallback to cluster runtimes.

### Environment

Kubernetes version:
```bash
$ kubectl version

```
Kubeflow Trainer version:
```bash
$ kubectl get pods -n kubeflow -l app.kubernetes.io/name=trainer -o jsonpath="{.items[*].spec.containers[*].image}"

```
Kubeflow Python SDK version:
```bash
$ pip show kubeflow

```

### Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

Contributor guide

Open the contributing guide

Research direction

Start in kubeflow/trainer/backends/kubernetes/backend.py around lines 174-223 and trace get_runtime's namespaced lookup, exception handling, and cluster-runtime fallback. Compare the behavior with the change from pull request #307. Done means get_runtime first tries namespaced runtimes and then successfully falls back to cluster runtimes without the lookup exception preventing retrieval.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.