kubeflow / kubeflow/trainer

Automatic PodDisruptionBudget for distributed TrainJobs

Open
#3,304 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.2k
Forks
1.1k
Avg merge
3d 22h
Merged PRs (30d)
39

Description

**What would you like to be added?**

When a distributed TrainJob is created, automatically create a `PodDisruptionBudget` (PDB) to protect training pods from voluntary disruptions (e.g. node drain, cluster autoscaler eviction).

For example, a PDB with `minAvailable` equal to the total number of training replicas would prevent any training pod from being evicted during cluster maintenance operations.

**Why is this needed?**

Distributed training is tightly coupled, all ranks must be available for training to make progress. When a training pod is evicted (e.g. during node drain), the remaining pods don't receive any signal and eventually crash due to communication timeouts, wasting compute time and requiring a full restart from the last checkpoint.

PDBs are the standard Kubernetes mechanism for protecting workloads from voluntary disruptions and would make distributed TrainJobs more resilient to routine cluster operations.

**Scope:** PDBs only apply to the [Eviction API](https://kubernetes.io/docs/concepts/scheduling-eviction/api-eviction/) and protect against voluntary disruptions like node drains and cluster autoscaler scale-downs. They do not protect against involuntary disruptions such as node failures, OOM kills, or hardware faults. They also do not interfere with Kueue preemption, which uses job suspension rather than pod eviction.

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by locating the TrainJob reconciliation entry point and the Kubernetes resource creation path, then review how distributed training replicas are represented. Done means a distributed TrainJob creates a PodDisruptionBudget with minAvailable equal to the total training replicas, with coverage for voluntary disruption behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
distributed-systems, infrastructure, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.