kubeflow / kubeflow/trainer

TrainJob status not updated when runtime resource reconciliation fails

Open
#3,671 1 comment 0 reactions 1 assignee Claimed by @AdeshDeshmukh View on GitHub
Dominant language
Go
Stars
2.2k
Forks
1.1k
Avg merge
3d 22h
Merged PRs (30d)
39

Description

**What happened?**:

When `reconcileObjects()` fails during the TrainJob reconciliation loop (either during `runtime.NewObjects()` or the SSA `client.Apply()`), the error is only emitted as a Kubernetes Event via `r.recorder.Eventf()`. The `TrainJob.Status.Conditions` are not updated to reflect the failure.

As a result, users running `kubectl get trainjob` see the job stuck in a non-terminal state (e.g., "Pending") with no indication of what went wrong. They must check the controller logs or ephemeral Events to diagnose the issue.

The code at `pkg/controller/trainjob_controller.go` explicitly has a TODO acknowledging this gap:
`// TODO (astefanutti): the error should be surfaced in the TrainJob status to indicate the creation of the runtime resources failed and the TrainJob is backed off until the next retry attempt.`

**What did you expect to happen?**:

When `reconcileObjects()` fails, the TrainJob status should be updated with a Failed condition containing the error message. This mirrors the existing behavior for unsupported runtimes (`TrainingRuntimeNotSupported`) and deadline exceeded (`DeadlineExceeded`).

**Environment**:
N/A — logic bug, not environment-dependent.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.