ZeRO skips the optimizer step on gradient overflow without logging anything

Repo: deepspeedai/DeepSpeed
Version: DeepSpeed 0.19.2

Problem

When ZeRO-1/2 detects a non-finite gradient it clears gradients and returns early from step()
(deepspeed/runtime/zero/stage_1_and_2.py:2154):

self._update_scale(self.overflow)
if self.overflow:
    see_memory_usage('After overflow before clearing gradients')
    self.zero_grad(set_to_none=True)
    ...
    return

Nothing is logged. Under fp16 this is usually survivable because _update_scale reports the loss
scale changing, which gives an operator an indirect signal. Under bf16 the loss scale is fixed,
so there is no output at all: engine.step() returns normally, the training loop advances, the LR
scheduler steps, metrics are emitted, and the run looks completely healthy while no weights are
ever updated.

DynamicLossScaler does log OVERFLOW! ... Skipping step, but CreateLossScaler only returns it
for fp16 + dynamic_scaling; bf16 always gets the static LossScaler, which logs nothing. The
engine does count the skip in engine.skipped_steps, but that is only surfaced by
_report_progress, which is gated on steps_per_print — and STEPS_PER_PRINT_DEFAULT is None,
so on a default config it never fires.

In a bf16 RL run this meant every optimizer step was skipped with no indication. The only
observable symptom was that weights synced downstream were byte-identical across steps; loss and
reward curves stayed plausible throughout, since reward noise readily hides a frozen policy.

Related

check_grad_overflow defaults to False for bf16 while fp16 forces it True
(deepspeed/runtime/precision_config.py). That default is defensible — bf16's range makes
overflow rare — but the combination is a sharp edge: with the check off, an Inf gradient flows
into the optimizer and writes NaN into every trainable tensor; with it on, steps are skipped
silently. Both failure modes are invisible. Documenting the flag's interaction with bf16 would
help even if the default stays.