google-deepmind / google-deepmind/optax
Adam-family optimizers (adam, amsgrad, adabelief, yogi) produce NaN on float16 zero-gradient steps
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 369
- Avg merge
- 10h 15m
- Merged PRs (30d)
- 7
Description
\`scale_by_adam\`, \`scale_by_amsgrad\`, \`scale_by_belief\`, and \`scale_by_yogi\` in \`optax/_src/transform.py\` compute their update as \`m / (sqrt(v + eps_root) + eps)\`. When params are \`float16\`, JAX's weak-type promotion casts the Python-float \`eps\` (default \`1e-8\`, or \`1e-16\` for belief) down to \`float16\` before the addition. float16's smallest representable subnormal is \`~6e-8\`, so \`eps\` rounds to exactly \`0.0\`.
On any step with an exactly-zero gradient (masked/padded tokens, a frozen-then-unfrozen layer, an unused embedding row, etc.), both moment estimates are \`0\`, so the denominator becomes \`sqrt(0) + 0 == 0\`, giving \`0 / 0 = NaN\`. Since NaN propagates through every later step, this silently poisons the rest of training.
\`float16\` is already an explicitly tested dtype elsewhere (\`alias_test.py::test_state_shape_dtype_shard_stability\`), but that test's gradient is never exactly zero, so it doesn't catch this. \`scale_by_yogi\`'s larger default \`eps=1e-3\` happens not to underflow, but is still fragile with a smaller \`eps\`.
Contributor guide
Research direction
Start in optax/_src/transform.py at scale_by_adam, scale_by_amsgrad, scale_by_belief, and scale_by_yogi, then inspect alias_test.py::test_state_shape_dtype_shard_stability for existing float16 coverage. Reproduce an exactly-zero-gradient float16 step and add regression coverage; done means these optimizers no longer produce NaN in that case while the existing dtype test remains valid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100