TrainJob status not updated when runtime resource reconciliation fails
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 1.1k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 39
Description
**What happened?**:
When `reconcileObjects()` fails during the TrainJob reconciliation loop (either during `runtime.NewObjects()` or the SSA `client.Apply()`), the error is only emitted as a Kubernetes Event via `r.recorder.Eventf()`. The `TrainJob.Status.Conditions` are not updated to reflect the failure.
As a result, users running `kubectl get trainjob` see the job stuck in a non-terminal state (e.g., "Pending") with no indication of what went wrong. They must check the controller logs or ephemeral Events to diagnose the issue.
The code at `pkg/controller/trainjob_controller.go` explicitly has a TODO acknowledging this gap:
`// TODO (astefanutti): the error should be surfaced in the TrainJob status to indicate the creation of the runtime resources failed and the TrainJob is backed off until the next retry attempt.`
**What did you expect to happen?**:
When `reconcileObjects()` fails, the TrainJob status should be updated with a Failed condition containing the error message. This mirrors the existing behavior for unsupported runtimes (`TrainingRuntimeNotSupported`) and deadline exceeded (`DeadlineExceeded`).
**Environment**:
N/A — logic bug, not environment-dependent.
Contributor guide
Assessment
This issue has not been assessed yet.