Support Multi-Node NVLink (MNNVL) for TrainJob with Topology Aware Scheduling
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 1.1k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 39
Description
Multi-Node NVLink (MNNVL) enables high-bandwidth GPU-to-GPU communication across nodes connected via NVSwitch fabric. For distributed training workloads to achieve best performance, all TrainJob pods should be placed on nodes within the same MNNVL domain.
Currently, users must manually create the compute domain CRD to leverage IMEX channels and manually configure placement constraints for their workloads. This doesn't scale.
We should add support for:
1. **Automatic compute domain creation** — Trainer should automatically create the compute domain CRD required for MNNVL and IMEX channel configuration, removing the manual setup burden from users.
2. **Topology Aware Scheduling (TAS) integration** — Use [TAS](https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/) to ensure best placement of TrainJob pods within MNNVL domains.
/kind feature
/area controller
CC @andreyvelich
Contributor guide
Research direction
Start by reading the TrainJob controller entry points and the Kueue Topology Aware Scheduling documentation linked in the issue. Trace how compute domain CRDs and IMEX channel configuration are currently handled, then define completion as automatic compute-domain creation and TAS placement of all TrainJob pods within one MNNVL domain.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- ai, backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100