kubeflow / kubeflow/trainer

TrainJob does not retry JobSet deletion after active deadline

Open
#3,833 4 comments 0 reactions 0 assignees View on GitHub
area/controller kind/bug
Dominant language
Go
Stars
2.2k
Forks
1.1k
Avg merge
3d 22h
Merged PRs (30d)
39

Description

### What happened?

When a TrainJob exceeds `spec.activeDeadlineSeconds`, the controller marks the TrainJob as failed and attempts to delete the associated JobSet. If that deletion returns a transient error, the error is only logged and reconciliation returns successfully:

https://github.com/kubeflow/trainer/blob/7c08264b93d3b83de45d1152b169858cadf1d8b3/pkg/controller/trainjob_controller.go#L155-L181

The `DeadlineExceeded` condition is preserved on later reconciles. Because `reconcileDeadline` exits early for an already-finished TrainJob, the failed JobSet deletion is never retried and the JobSet workload may continue running past the configured deadline.

A focused reproduction is:

1. Reconcile a TrainJob whose active deadline has elapsed.
2. Make the JobSet DELETE request return a transient API error.
3. Observe that the TrainJob is patched with the terminal `DeadlineExceeded` condition.
4. Reconcile again and observe that JobSet deletion is skipped because the TrainJob is already considered finished.

### What did you expect to happen?

A transient deletion failure should be surfaced and retried until the JobSet is deleted or the API returns NotFound, while preserving the terminal deadline condition.

A possible focused fix would keep deadline-exceeded TrainJobs eligible for cleanup retries and add a regression test using a client that fails the first deletion attempt.

### Environment

Kubeflow Trainer version: current master at `7c08264b93d3b83de45d1152b169858cadf1d8b3`

Kubernetes version: N/A (confirmed from controller control flow; regression can be covered with a fake client)

Kubeflow Python SDK version: N/A

### Contribution

I'd like to work on this and submit a focused PR with regression coverage once maintainers confirm the expected retry behavior.

_AI assistance disclosure: Codex assisted with repository inspection and technical drafting; I verified the reported behavior against the current source._

Contributor guide

Open the contributing guide

Research direction

Start in pkg/controller/trainjob_controller.go at reconcileDeadline and the linked lines handling JobSet deletion. Add focused regression coverage with a fake client that fails the first DELETE, then reconcile again. Done means transient deletion errors are retried, NotFound is accepted, and the terminal DeadlineExceeded condition remains preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.