bug(trainer): list_jobs and get_job fail after a referenced runtime is deleted
- Dominant language
- Python
- Stars
- 148
- Forks
- 262
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 1
Description
### What happened?
In the Kubernetes backend, `list_jobs()` and `get_job()` look up each TrainJob's runtime live with `get_runtime()`. If that runtime has been deleted, the lookup raises and `list_jobs()` fails with "Failed to list TrainJobs", so one such job breaks the whole list.
Trainer v2.3.0 removed the runtime finalizers (kubeflow/trainer#3716) and snapshots the runtime per job (KEP-2599), so deleting a runtime while its jobs still exist is now expected.
Found in the code (`kubeflow/trainer/backends/kubernetes/backend.py`, `__get_trainjob_from_cr`), not run on a cluster.
### What did you expect to happen?
`list_jobs()` still returns the other jobs, and `get_job()` works for a job whose runtime is gone.
Should these jobs be skipped, returned without a runtime, or read from the runtime snapshot? I'll open a PR once there's a direction.
### Environment
Kubernetes version: n/a, found in code
Kubeflow Trainer version: v2.3.0
Kubeflow Python SDK version: 0.5.0 and main (3febcd7)
Contributor guide
Research direction
Start in kubeflow/trainer/backends/kubernetes/backend.py, especially __get_trainjob_from_cr and the list_jobs() and get_job() entry points. Read how get_runtime() is used and review the referenced runtime snapshot behavior in KEP-2599. Confirm with maintainers whether missing runtimes should be skipped, omitted, or reconstructed from snapshots; done means the agreed behavior works without one deleted runtime breaking other job lookups.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100