NVIDIA-NeMo / NVIDIA-NeMo/Emerging-Optimizers

Docs: note where library defaults diverge from the arXiv 2607.20548 recipe (momentum, weight decay, NS iterations, RMS scale)

Open Beginner friendly
#259 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
274
Forks
51
Avg merge
1d 3h
Merged PRs (30d)
6

Description

The recent paper "SOAP, Muon, and Beyond" (arXiv 2607.20548) points readers to this repository as its released code. Readers who install the library and use Muon's constructor defaults get a materially different optimizer than the paper's experiments on four axes:

axis paper recipe library default (muon.py @ HEAD)
momentum 0.9 0.95
decoupled weight decay 0.1 0.01
orthogonalization 16-step PolarExpress 5-step quintic
update-RMS correction ~0.2 (sqrt((1-b1)/(1+b1))) extra_scale_factor=1.0 (off)

Since the paper itself argues optimizer comparisons are uninterpretable without matched update RMS, a short README note or a checked-in "paper recipe" reference config would prevent exactly the silent config mismatch the paper warns about. The only current breadcrumb is the extra_scale_factor docstring's optional 0.2 mention, which cites arXiv 2502.16982 rather than the recipe. Happy to PR the table above into the README if that's welcome.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the Muon constructor defaults in muon.py with the proposed table and the existing extra_scale_factor docstring. Review the README structure and the cited paper context, then add a concise note or checked-in paper recipe reference covering the four divergences and their sources. Done means readers can identify the paper settings separately from the library defaults.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.