why use for all-reduce when `contiguous_gradients` is False?

Hi there,

On single node with 8A100 GPUs connected with nvlinks, I can still observe the performance benefits with contiguous_gradients set to False. e.g., 10B model backward time: ~1329.30ms vs ~ 1593.24ms (contiguous_gradients=False vs. contiguous_gradients=True).

I am wondering where is the benefits coming from. The all-reduce suppose to incur more communication overheads than reduce-scatter operation.

Could please explain a littile bit about why use all-reduce for each gradient tensor? (I not completely sure that zero3 will do all-reduce for all gradient tensors. But base on the code path at here, I guess so)
https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/runtime/zero/stage3.py#L2314

And what could be the overheads introduced with setting contiguous_gradients=True?

Thanks!

Start at deepspeed/runtime/zero/stage3.py around line 2314 and trace the gradient communication path for both contiguous_gradients settings. Compare the all-reduce and reduce-scatter behavior and identify the overheads of contiguous gradient buffers. Done means documenting a clear explanation grounded in that code path and the reported timings.