deepspeedai / deepspeedai/DeepSpeed
[BUG] The different lr for Muon doesn't work?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
I highly appreciate the improvements you've made in supporting the Muon optimizer. However, I've run into an issue when trying to set the individual muon_lr and adam_lr within the configuration JSON.
I've found that the optimizer still defaults to setting a single lr for both the Muon parameter group and the Adam parameter group. I've verified that I am on the latest commit, as the relevant processing code for this feature branch is present in my local engine.py.
All other settings in the configuration are working as expected, but the specific muon_lr and adam_lr settings are not taking effect. Has anyone else encountered this issue?
"optimizer": {
"type": "Muon",
"params": {
"lr": 1e-4,
"betas": [0.9, 0.95],
"eps": 1e-8,
"weight_decay": 0.01,
"muon_lr": 2e-2,
"adam_lr": 1e-5
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in engine.py at the Muon configuration processing mentioned in the report, then reproduce the supplied JSON configuration and inspect the resulting Muon and Adam parameter-group learning rates. Done means muon_lr and adam_lr are applied independently instead of both groups falling back to lr, with a regression test covering the configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100