[Question] Duplicate data parallel group names with mixture of experts (MoE) implementation

I am using the deepspeed MoE layer (https://deepspeed.readthedocs.io/en/latest/moe.html) to train a GPT2 model from scratch. I am using the following zero2 config yaml for training the model on 8 GPUs (1 node).

{
    "bf16": {"enabled": false},
    "prescale_gradients": false,
    "zero_optimization": {
        "stage": 2,
        "contiguous_gradients": true,
        "allgather_partitions": true,
        "reduce_scatter": true,
        "overlap_comm": true
    },
    "zero_allow_untested_optimizer": true,
    "allgather_bucket_size": 5e8,
    "reduce_bucket_size": 5e8,
    "cpu_checkpointing": false,
}

I initialize the model with the following deepspeed context

context = deepspeed.zero.Init(
        remote_device=trainer.strategy.remote_device,
        pin_memory=True,
        config=config,
        dtype=th.float32,
        enabled=False
)

with context:
    model = GPT2LMHeadModel(model_config)

where I modified the GPT2LMHeadModel class to include the MoE layer with num_experts set to 16, ep_size set to 8. Deepspeed initializes the model to create multiple data parallel expert groups with the same name - ep_size_8 which is later creating an issue due to duplicate parameter group names, when I want to monitor the AdamW optimizer learning rate separated by groups.

I see the naming scheme designed here - https://github.com/microsoft/DeepSpeed/blob/0636c74c5e27757d48f64f33f330d7bb975fc5a8/deepspeed/utils/groups.py#L134/ . Is there a reason to name all groups the same or is this an artifact of my implementation? I am using deepspeed 0.10.0 version.

Start with deepspeed/utils/groups.py around the naming scheme linked in the issue, then compare how deepspeed/moe/layer.py creates expert groups with ep_size=8 and num_experts=16. Reproduce the supplied 8-GPU ZeRO-2 configuration and inspect the resulting AdamW parameter-group names; done means determining whether duplicate ep_size_8 names are expected or an implementation issue.