[BUG] pipeline parallelism+fp16+moe isn't working

Describe the bug
My model use deepspeed PipelineModule(num_stages=4) split into 4 parts, and my deepspeed.moe.layer.MoE is only set in the pipeline stage1 layer. When my model train_batch, the program will get stuck, the specific issue occurs in FP16_Optimizer step.

Here is our deepspeed config

{
   "train_batch_size": 4,
   "train_micro_batch_size_per_gpu" 1,
   "fp16": {
      "enabled": true,
      "auto_cast": true
   },
   "optimizer": {
      "type": "AdamW",
      "params": {
         "lr": 0.001,
         "betas": [
            0.9,
            0.95
         ],
         "weight_decay": 0.05
      }
   },
   "zero_optimization": {
      "stage": 0
   }
}

Source code with issues
my pipeline_parallel_world_size is 4, the code will enter the following branch, but my moe layer only is set in pipeline stage1, then all_reduce will make program stuck. If I delete this code, it will run successfully.

https://github.com/microsoft/DeepSpeed/blob/10ba3dde84d00742f3635c48db09d6eccf0ec8bb/deepspeed/runtime/utils.py#L892-L893

I don't know why all_reduce needs to be done here, it doesn't seem meaningful

Start with deepspeed/runtime/utils.py around lines 892-893 and trace the FP16_Optimizer step reached by train_batch. Reproduce the configuration with PipelineModule(num_stages=4), FP16, and MoE only in pipeline stage 1, then inspect the all_reduce participation across stages. Done means the hang is explained and the behavior is corrected or clearly documented with a validating reproduction.