kubeflow / kubeflow/sdk

Checking for the image in the manifest of TrainingRuntime

Open
#468 4 comments 0 reactions 0 assignees View on GitHub
kind/bug needs-triage
Dominant language
Python
Stars
148
Forks
262
Avg merge
1d 2h
Merged PRs (30d)
1

Description

### What happened?

Now there is a check for the image in the TrainingRuntime manifest, although the spec itself does not require this. We want to use a single TrainingRuntime manifest to run multiple images that use the same resources using CustomTrainer or CustomTrainerContainer, which contain the image attribute.
https://github.com/kubeflow/sdk/blob/0.4.0/kubeflow/trainer/backends/kubernetes/utils.py#L106

### What did you expect to happen?

The image check will be performed when creating Trainer CR.
https://github.com/kubeflow/sdk/blob/0.4.0/kubeflow/trainer/backends/kubernetes/utils.py#L426

### Environment

Kubernetes version:
```bash
$ kubectl version
Client Version: v1.35.3
Kustomize Version: v5.7.1
Server Version: v1.33.0
Warning: version difference between client (1.35) and server (1.33) exceeds the supported minor version skew of +/-1
```
Kubeflow Trainer version:
```bash
$ kubectl get pods -n kubeflow -l app.kubernetes.io/name=trainer -o jsonpath="{.items[*].spec.containers[*].image}"

```
Kubeflow Python SDK version:
```bash
$ pip show kubeflow
0.4.0
```

### Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

Contributor guide

Open the contributing guide

Research direction

Start in kubeflow/trainer/backends/kubernetes/utils.py, comparing the image check near line 106 with the Trainer CR creation path near line 426. Trace when TrainingRuntime and CustomTrainer or CustomTrainerContainer manifests are validated; done means image validation occurs when creating the Trainer CR, not because TrainingRuntime itself requires an image.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.