TrainJob may be recreated after a failed JobSet is deleted
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 1.1k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 39
Description
### What happened?
A terminally failed TrainJob may be restarted after its JobSet is deleted, either manually or by the JobSet TTL controller.
This can cause a failed training workload to run again unexpectedly and consume additional cluster resources.
### What did you expect to happen?
A terminally failed TrainJob should remain failed after its JobSet is deleted. Runtime objects should not be recreated unless an explicitly supported retry or recovery operation is requested.
### Environment
None
### Impacted by this bug?
Give it a 👍 We prioritize the issues with most 👍
Contributor guide
Research direction
No files or tests are named. Start by tracing the TrainJob and JobSet reconciliation paths and reproduce deletion of a JobSet after terminal failure. Done means a terminally failed TrainJob stays failed and its runtime objects are not recreated without an explicitly supported retry or recovery operation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100