MFSDP v2 Muon: Localize optimizer-step communication within NVLink domains
Open
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
Follow-up to #5533.
Refine parameter placements during forward and backward so that the Muon optimizer step only needs to communicate within a node or NVLink domain. See http://nv/kOb for more details.
This should reduce cross-node communication in the optimizer path while preserving MFSDP v2’s intended training behavior.
cc @janEbert
Contributor guide
Research direction
Start with follow-up issue #5533 and the linked NVLink-domain details. Trace MFSDP v2 parameter placement through forward and backward, then examine the Muon optimizer step. Done means optimizer-step communication stays within a node or NVLink domain while intended MFSDP v2 training behavior is preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100