Zero stage 2 : contiguous_gradients and "Reduce"

I am running the 8B model as described in Table 1 of the Zero paper. on 8 GPUs.
I notice that the "contiguous_gradients" setting in the config seems to control whether "Reduce" or "Allreduce" is used for gradient reduction in backward pass. That is, I see the following in the NCCL Debug log only when "contiguous_gradients" is "true":

I see this being referenced in #264 but it wasn't clear why "contiguous_gradients" should control the communication pattern. As the answer in #264 mentioned it should only "defragment the memory during backward propagation".

Start by tracing the contiguous_gradients configuration through the backward-pass gradient reduction path and compare the NCCL Debug output for true and false. Use the Zero stage 2 setup from Table 1 on 8 GPUs, then document whether the communication difference is expected and how it relates to the memory-defragmentation behavior described in #264.