NVIDIA / NVIDIA/Megatron-LM

MFSDP v2 Muon: Localize optimizer-step communication within NVLink domains

Open
#6,603 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

Follow-up to #5533.

Refine parameter placements during forward and backward so that the Muon optimizer step only needs to communicate within a node or NVLink domain. See http://nv/kOb for more details.

This should reduce cross-node communication in the optimizer path while preserving MFSDP v2’s intended training behavior.

cc @janEbert

Contributor guide

Open the contributing guide

Research direction

Start with follow-up issue #5533 and the linked NVLink-domain details. Trace MFSDP v2 parameter placement through forward and backward, then examine the Muon optimizer step. Done means optimizer-step communication stays within a node or NVLink domain while intended MFSDP v2 training behavior is preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.