google-deepmind / google-deepmind/optax
Implementing AdaMuon
Open
type:feature
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 369
- Avg merge
- 10h 15m
- Merged PRs (30d)
- 7
Description
This is a new optimizer, that combines elementwise adaptivity with orthogonal updates for large networks
Novelty:
1. an elementwise second momentum estimator applied to update directions
2. a sign-stabilized update where the momentum is first sign-transformed before orthogonalization
Paper link : https://arxiv.org/abs/2507.11005
Contributor guide
Assessment
This issue has not been assessed yet.