ClusterRuntime un-getable
- Dominant language
- Python
- Stars
- 148
- Forks
- 262
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 1
Description
### What happened?
It looks like with recent changes to the Kubernetes backend, ClusterRuntime are not retrievable.
When attempting to get a ClusterRuntime, it can never go past the exception caught on L199:
https://github.com/kubeflow/sdk/blob/db76de0ca8c32540b8a177a713e73f41c1bf81af/kubeflow/trainer/backends/kubernetes/backend.py#L174-L223
The change seems to come from https://github.com/kubeflow/sdk/pull/307. Based on skimming that PR, it looks like it probably worked originally with the `pass` but that was pulling out due to a suggestion from copilot. It doesn't need to raise the error there as that requires something above it to catch it. It should probably just print/log it instead of catching and raising again.
Disclaimer: I am not familiar with this codebase nor have I had time to fully dig into it, so maybe I missed up something in my configuration. Either way, it warrants a deeper look.
### What did you expect to happen?
`get_runtime` should first try namespaced runtimes and then fallback to cluster runtimes.
### Environment
Kubernetes version:
```bash
$ kubectl version
```
Kubeflow Trainer version:
```bash
$ kubectl get pods -n kubeflow -l app.kubernetes.io/name=trainer -o jsonpath="{.items[*].spec.containers[*].image}"
```
Kubeflow Python SDK version:
```bash
$ pip show kubeflow
```
### Impacted by this bug?
Give it a 👍 We prioritize the issues with most 👍
Contributor guide
Research direction
Start in kubeflow/trainer/backends/kubernetes/backend.py around lines 174-223 and trace get_runtime's namespaced lookup, exception handling, and cluster-runtime fallback. Compare the behavior with the change from pull request #307. Done means get_runtime first tries namespaced runtimes and then successfully falls back to cluster runtimes without the lookup exception preventing retrieval.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100