kubeflow / kubeflow/trainer

Support Multi-Node NVLink (MNNVL) for TrainJob with Topology Aware Scheduling

Open
#3,264 14 comments 3 reactions 0 assignees View on GitHub
area/controller kind/feature lifecycle/stale
Dominant language
Go
Stars
2.2k
Forks
1.1k
Avg merge
3d 22h
Merged PRs (30d)
39

Description

Multi-Node NVLink (MNNVL) enables high-bandwidth GPU-to-GPU communication across nodes connected via NVSwitch fabric. For distributed training workloads to achieve best performance, all TrainJob pods should be placed on nodes within the same MNNVL domain.

Currently, users must manually create the compute domain CRD to leverage IMEX channels and manually configure placement constraints for their workloads. This doesn't scale.

We should add support for:

1. **Automatic compute domain creation** — Trainer should automatically create the compute domain CRD required for MNNVL and IMEX channel configuration, removing the manual setup burden from users.

2. **Topology Aware Scheduling (TAS) integration** — Use [TAS](https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/) to ensure best placement of TrainJob pods within MNNVL domains.

/kind feature
/area controller

CC @andreyvelich

Contributor guide

Open the contributing guide

Research direction

Start by reading the TrainJob controller entry points and the Kueue Topology Aware Scheduling documentation linked in the issue. Trace how compute domain CRDs and IMEX channel configuration are currently handled, then define completion as automatic compute-domain creation and TAS placement of all TrainJob pods within one MNNVL domain.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
ai, backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.