[BUG]Loss connection to ranks

Describe the bug
When I'm training my model with 16 gpu(2node), the backward will be stucked in

accelerate.utils.deepspeed line 270
self.engine.step()

So i step in and find out that in
deepspeed.runtime.stage_1_and_2.py line 2098

def step(self, closure=None):
        """
        Not supporting closure.
        """
        self.micro_step_id = INITIAL_MICRO_STEP_ID

        see_memory_usage("In step before checking overflow")

        # First compute norm for all group so we know if there is overflow
        if self.check_grad_overflow:
            self.check_overflow(partition_gradients=self.partition_gradients)

        prev_scale = self.loss_scale
        self._update_scale(self.overflow)
        if self.overflow:
            see_memory_usage('After overflow before clearing gradients')
            self.zero_grad(set_to_none=True)
            if self.cpu_offload:
                self.reset_cpu_buffers()
            else:
                for k in self.averaged_gradients.keys():
                    self.averaged_gradients[k] = None
                    self.all_grad_tensors[k] = None

            see_memory_usage('After overflow after clearing gradients')

            for timer in OPTIMIZER_TIMERS:
                self.timers(timer).start()
                self.timers(timer).stop()
            return

        # Step 1:- Calculate gradient norm using bit-16 grads
        see_memory_usage('Before norm calculation')
        scaled_global_grad_norm = self.scaled_global_norm()
        self._global_grad_norm = scaled_global_grad_norm / prev_scale
        see_memory_usage('After norm before optimizer')

In step 1, scaled_global_grad_norm = self.scaled_global_norm()

def scaled_global_norm(self, norm_type=2):
        assert norm_type == 2, "only L2 norm supported"
        norm_groups = []
        for i, group in enumerate(self.bit16_groups):
            if self.cpu_offload:
                norm = self.complete_grad_norm_calculation_for_cpu_offload(self.params_in_partition[i])
                norm_groups.append(norm)
            else:
                norm_groups.append(self.get_grad_norm_direct(self.averaged_gradients[i], self.params_in_partition[i]))

        if self.has_moe_layers:
            self._average_expert_grad_norms(norm_groups)

        # calculating L2 norm
        return torch.linalg.vector_norm(torch.stack(norm_groups), ord=norm_type)

I use pdb here, output dist.get_rank(), i can get all rank numbers(0~15), however, if i use command
pirnt(self.bit16_groups), it will only give 14 results, and I will loss connections. I don't know what triggered this.

To Reproduce

1. add SAM3 to any MLLMs, just give the last hidden_states to SAM3 video predictor(use projector)
2. start training with 2nodes(16 gpus)
3. Note: I can't reproduce this in 1 node or 4 nodes

Expected behavior
stuck at the 9th step

ds_report output

Screenshots

System info (please complete the following information):

• GPU count and types [16 A100s(2 nodes)]
• Python version 3.10.4

Launcher context
using transformers trainer

Start by reproducing the Transformers Trainer run with 16 A100s across two nodes, then inspect accelerate.utils.deepspeed at line 270 and deepspeed.runtime.stage_1_and_2.py around step() and scaled_global_norm(). Compare dist.get_rank() with the length of bit16_groups across ranks, including the one-node and four-node cases. Done means identifying why two ranks or parameter groups disappear and documenting a reliable fix or diagnostic.