google-deepmind / google-deepmind/optax

Adam-family optimizers (adam, amsgrad, adabelief, yogi) produce NaN on float16 zero-gradient steps

Open
#1,754 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Python
Stars
2.3k
Forks
369
Avg merge
10h 15m
Merged PRs (30d)
7

Description

\`scale_by_adam\`, \`scale_by_amsgrad\`, \`scale_by_belief\`, and \`scale_by_yogi\` in \`optax/_src/transform.py\` compute their update as \`m / (sqrt(v + eps_root) + eps)\`. When params are \`float16\`, JAX's weak-type promotion casts the Python-float \`eps\` (default \`1e-8\`, or \`1e-16\` for belief) down to \`float16\` before the addition. float16's smallest representable subnormal is \`~6e-8\`, so \`eps\` rounds to exactly \`0.0\`.

On any step with an exactly-zero gradient (masked/padded tokens, a frozen-then-unfrozen layer, an unused embedding row, etc.), both moment estimates are \`0\`, so the denominator becomes \`sqrt(0) + 0 == 0\`, giving \`0 / 0 = NaN\`. Since NaN propagates through every later step, this silently poisons the rest of training.

\`float16\` is already an explicitly tested dtype elsewhere (\`alias_test.py::test_state_shape_dtype_shard_stability\`), but that test's gradient is never exactly zero, so it doesn't catch this. \`scale_by_yogi\`'s larger default \`eps=1e-3\` happens not to underflow, but is still fragile with a smaller \`eps\`.

Contributor guide

Open the contributing guide

Research direction

Start in optax/_src/transform.py at scale_by_adam, scale_by_amsgrad, scale_by_belief, and scale_by_yogi, then inspect alias_test.py::test_state_shape_dtype_shard_stability for existing float16 coverage. Reproduce an exactly-zero-gradient float16 step and add regression coverage; done means these optimizers no longer produce NaN in that case while the existing dtype test remains valid.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.