kubeflow / kubeflow/sdk

bug(trainer): list_jobs and get_job fail after a referenced runtime is deleted

Open
#797 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
148
Forks
262
Avg merge
1d 2h
Merged PRs (30d)
1

Description

### What happened?

In the Kubernetes backend, `list_jobs()` and `get_job()` look up each TrainJob's runtime live with `get_runtime()`. If that runtime has been deleted, the lookup raises and `list_jobs()` fails with "Failed to list TrainJobs", so one such job breaks the whole list.

Trainer v2.3.0 removed the runtime finalizers (kubeflow/trainer#3716) and snapshots the runtime per job (KEP-2599), so deleting a runtime while its jobs still exist is now expected.

Found in the code (`kubeflow/trainer/backends/kubernetes/backend.py`, `__get_trainjob_from_cr`), not run on a cluster.

### What did you expect to happen?

`list_jobs()` still returns the other jobs, and `get_job()` works for a job whose runtime is gone.

Should these jobs be skipped, returned without a runtime, or read from the runtime snapshot? I'll open a PR once there's a direction.

### Environment

Kubernetes version: n/a, found in code
Kubeflow Trainer version: v2.3.0
Kubeflow Python SDK version: 0.5.0 and main (3febcd7)

Contributor guide

Open the contributing guide

Research direction

Start in kubeflow/trainer/backends/kubernetes/backend.py, especially __get_trainjob_from_cr and the list_jobs() and get_job() entry points. Read how get_runtime() is used and review the referenced runtime snapshot behavior in KEP-2599. Confirm with maintainers whether missing runtimes should be skipped, omitted, or reconstructed from snapshots; done means the agreed behavior works without one deleted runtime breaking other job lookups.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.