google-deepmind / google-deepmind/optax

Implementing AdaMuon

Open
#1,575 4 comments 1 reaction 0 assignees View on GitHub
type:feature
Dominant language
Python
Stars
2.3k
Forks
369
Avg merge
10h 15m
Merged PRs (30d)
7

Description

This is a new optimizer, that combines elementwise adaptivity with orthogonal updates for large networks

Novelty:
1. an elementwise second momentum estimator applied to update directions
2. a sign-stabilized update where the momentum is first sign-transformed before orthogonalization

Paper link : https://arxiv.org/abs/2507.11005

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.