Automatic PodDisruptionBudget for distributed TrainJobs
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 1.1k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 39
Description
**What would you like to be added?**
When a distributed TrainJob is created, automatically create a `PodDisruptionBudget` (PDB) to protect training pods from voluntary disruptions (e.g. node drain, cluster autoscaler eviction).
For example, a PDB with `minAvailable` equal to the total number of training replicas would prevent any training pod from being evicted during cluster maintenance operations.
**Why is this needed?**
Distributed training is tightly coupled, all ranks must be available for training to make progress. When a training pod is evicted (e.g. during node drain), the remaining pods don't receive any signal and eventually crash due to communication timeouts, wasting compute time and requiring a full restart from the last checkpoint.
PDBs are the standard Kubernetes mechanism for protecting workloads from voluntary disruptions and would make distributed TrainJobs more resilient to routine cluster operations.
**Scope:** PDBs only apply to the [Eviction API](https://kubernetes.io/docs/concepts/scheduling-eviction/api-eviction/) and protect against voluntary disruptions like node drains and cluster autoscaler scale-downs. They do not protect against involuntary disruptions such as node failures, OOM kills, or hardware faults. They also do not interfere with Kueue preemption, which uses job suspension rather than pod eviction.
Contributor guide
Research direction
No files or tests are named. Start by locating the TrainJob reconciliation entry point and the Kubernetes resource creation path, then review how distributed training replicas are represented. Done means a distributed TrainJob creates a PodDisruptionBudget with minAvailable equal to the total training replicas, with coverage for voluntary disruption behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- distributed-systems, infrastructure, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100