kubeflow / kubeflow/sdk

bug(trainer): LocalProcess and Container backends raise ValueError instead of RuntimeError for job-not-found

Open
#433 2 comments 0 reactions 0 assignees View on GitHub
area/local kind/bug
Dominant language
Python
Stars
148
Forks
262
Avg merge
1d 2h
Merged PRs (30d)
1

Description

### What happened?

`TrainerClient.get_job()`, `delete_job()`, `get_job_logs()`, and `wait_for_job_status()` document `RuntimeError` for operational failures like job-not-found. The Kubernetes backend follows this contract, but LocalProcess and Container backends raise `ValueError` instead.

This means callers writing `except RuntimeError` to handle missing jobs will not catch the actual exception when using local or container backends.

### Steps to Reproduce

```python
from kubeflow.trainer.backends.localprocess.backend import LocalProcessBackend

class Cfg:
auto_remove = False
execution_dir = "/tmp"

backend = LocalProcessBackend(Cfg())

try:
backend.get_job("nonexistent")
except RuntimeError:
print("caught") # never reached
except ValueError as e:
print(f"BUG: {e}") # "No TrainJob with name nonexistent"
```

Same happens with `delete_job`, `get_job_logs`, `wait_for_job_status`.

### Affected locations

**LocalProcess backend** (`localprocess/backend.py`):
- `get_job` (line 172): `ValueError("No TrainJob with name {name}")`
- `get_job_logs` (line 201): same
- `wait_for_job_status` (line 227): same
- `delete_job` (line 256): same
- `get_runtime` (line 60): `ValueError("Runtime '{name}' not found.")`

**Container backend** (`container/backend.py`):
- `_get_job_containers` (line 509): `ValueError("No TrainJob with name {name}")`
- `__get_trainjob_from_containers` (lines 669, 674, 678, 687, 690): various `ValueError` for metadata/network/runtime lookup failures

**Kubernetes backend**: correctly raises `RuntimeError` for all of these.

### What did you expect to happen?

Operational failures (job not found, runtime not found) should raise `RuntimeError` consistently across all backends, matching the documented API contract.

Input validation errors (e.g. "CustomTrainer must be set", "polling_interval >= timeout") should remain `ValueError`.

### Proposed Fix

Change `raise ValueError("No TrainJob with name ...")` to `raise RuntimeError(...)` in the affected methods across both backends. Keep `ValueError` for actual input validation.

/kind bug
/area local

Contributor guide

Open the contributing guide

Research direction

Read localprocess/backend.py and container/backend.py at the listed job and runtime lookup methods, then compare their exceptions with the Kubernetes backend. Verify that operational lookup failures raise RuntimeError while input validation remains ValueError, and confirm the documented TrainerClient behavior across both backends.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.