[BUG] fp16 optimizers: printing log info on all ranks

These get printed to all ranks:

./deepspeed/runtime/fp16/unfused_optimizer.py:                            f"No Grad overflow for {self.scale_window} iterations")
./deepspeed/runtime/fp16/fused_optimizer.py:                            f"No Grad overflow for {self.scale_window} iterations")
./deepspeed/runtime/fp16/unfused_optimizer.py:                            f"Increasing dynamic loss scale from {prev_scale} to {self.cur_scale}"
./deepspeed/runtime/fp16/fused_optimizer.py:                            f"Increasing dynamic loss scale from {prev_scale} to {self.cur_scale}"

so hundreds of these get printed on a many node setup.

can we please make these print on global rank 0 only?

update: there are other places as well:

[2021-11-08 18:30:01,688] [INFO] [unfused_optimizer.py:199:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 65536.0, reducing to 32768.0
[2021-11-08 18:30:01,688] [INFO] [unfused_optimizer.py:275:_update_scale] Grad overflow on iteration: 2983
[2021-11-08 18:30:01,688] [INFO] [unfused_optimizer.py:276:_update_scale] Reducing dynamic loss scale from 65536.0 to 32768.0

so it's probably in many places...

Thank you!

Start with deepspeed/runtime/fp16/unfused_optimizer.py and deepspeed/runtime/fp16/fused_optimizer.py, inspecting the listed dynamic loss-scale logging sites and nearby overflow handling. Search the fp16 optimizer code for other related log messages, then verify in distributed training that these messages are emitted only on global rank 0 while optimizer behavior remains unchanged.