[BUG] Memory increase consistently when using multiple NCCL_IB_HCA.

My cluster have multiple ib interfaces (8xHDR).When using multiple NCCL_IB_HCA in enviroment

export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7

The communation speed didn't improve, however, the GPU memory increased a lot, compared with only two interfaces, i.e., NCCL_IB_HCA=mlx5_0,mlx5_1. This issue is consistent with the number of IB interfaces I use: the more the IB interfaces, the larger memory consumption.

I use the zero-3 for training. the ds_config is shown bellow:

{
"gradient_accumulation_steps": 1,
"train_micro_batch_size_per_gpu": 24,
"prescale_gradients": false,
"zero_allow_untested_optimizer": true,
"optimizer": {
"type": "AdamW",
"params": {
"lr": 1e-8,
"eps": 1.0e-8,
"betas": [
0.9,
0.98
],
"weight_decay": 0.1
}
},
"flops_profiler": {
"enabled": true,
"profile_step": 3,
"module_depth": -1,
"top_modules": 1,
"detailed": true,
"output_file": null
},
"scheduler": {
"type": "WarmupDecayLR",
"params": {
"total_num_steps": 150000,
"warmup_min_lr": 0,
"warmup_max_lr": 1e-5,
"warmup_num_steps": 200
}
},
"zero_optimization": {
"stage":3,
"contiguous_gradients": false,
"reduce_bucket_size": 7.96e7,
"overlap_comm": true,
"reduce_scatter": true,
"stage3_max_live_parameters" : 2e8,
"stage3_max_reuse_distance" : 2e8,
"stage3_prefetch_bucket_size" : 1e8,
"stage3_param_persistence_threshold" : 1e6,
"sub_group_size" : 8e8,
"stage3_gather_16bit_weights_on_model_save": true,
"offload_param": {
"device": "cpu",
"pin_memory": true
}
},
"steps_per_print": 5,
"gradient_clipping": 1.0,
"wall_clock_breakdown": true,
"bf16": {
"enabled": true
}
}

and deepspeed_env is set by:

OMPI_ALLOW_RUN_AS_ROOT=1
NCCL_DEBUG=info
NCCL_LAUNCH_MODE=PARALLEL
NCCL_CHECKS_DISABLE=1
NCCL_IB_DISABLE=0
NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7
NCCL_IB_CUDA_SUPPORT=1

No files, tests, or code entry points are named. Start by reproducing the DeepSpeed ZeRO-3 training setup with NCCL_IB_HCA set to two interfaces and then eight, comparing GPU memory consumption and communication speed; done means isolating and documenting the cause of the interface-count-dependent memory increase.