bug(trainer): LocalProcess and Container backends raise ValueError instead of RuntimeError for job-not-found
- Dominant language
- Python
- Stars
- 148
- Forks
- 262
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 1
Description
### What happened?
`TrainerClient.get_job()`, `delete_job()`, `get_job_logs()`, and `wait_for_job_status()` document `RuntimeError` for operational failures like job-not-found. The Kubernetes backend follows this contract, but LocalProcess and Container backends raise `ValueError` instead.
This means callers writing `except RuntimeError` to handle missing jobs will not catch the actual exception when using local or container backends.
### Steps to Reproduce
```python
from kubeflow.trainer.backends.localprocess.backend import LocalProcessBackend
class Cfg:
auto_remove = False
execution_dir = "/tmp"
backend = LocalProcessBackend(Cfg())
try:
backend.get_job("nonexistent")
except RuntimeError:
print("caught") # never reached
except ValueError as e:
print(f"BUG: {e}") # "No TrainJob with name nonexistent"
```
Same happens with `delete_job`, `get_job_logs`, `wait_for_job_status`.
### Affected locations
**LocalProcess backend** (`localprocess/backend.py`):
- `get_job` (line 172): `ValueError("No TrainJob with name {name}")`
- `get_job_logs` (line 201): same
- `wait_for_job_status` (line 227): same
- `delete_job` (line 256): same
- `get_runtime` (line 60): `ValueError("Runtime '{name}' not found.")`
**Container backend** (`container/backend.py`):
- `_get_job_containers` (line 509): `ValueError("No TrainJob with name {name}")`
- `__get_trainjob_from_containers` (lines 669, 674, 678, 687, 690): various `ValueError` for metadata/network/runtime lookup failures
**Kubernetes backend**: correctly raises `RuntimeError` for all of these.
### What did you expect to happen?
Operational failures (job not found, runtime not found) should raise `RuntimeError` consistently across all backends, matching the documented API contract.
Input validation errors (e.g. "CustomTrainer must be set", "polling_interval >= timeout") should remain `ValueError`.
### Proposed Fix
Change `raise ValueError("No TrainJob with name ...")` to `raise RuntimeError(...)` in the affected methods across both backends. Keep `ValueError` for actual input validation.
/kind bug
/area local
Contributor guide
Research direction
Read localprocess/backend.py and container/backend.py at the listed job and runtime lookup methods, then compare their exceptions with the Kubernetes backend. Verify that operational lookup failures raise RuntimeError while input validation remains ValueError, and confirm the documented TrainerClient behavior across both backends.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100