A possible solution to resolve the issue of deepspeed.initialize() hanging.

First of all, I would like to express my deep gratitude to this team for their great contribution, which allows us to fine-tune large models in a relatively easy manner.

This article describes the method I successfully used to resolve the issue of deepspeed.initialize() hanging. After a simple search, I noticed that it seems no one has mentioned this before, so I decided to leave some traces here for future reference for others.

Okay, the core idea is, if you are also using a container, please check the size of the MTU (Maximum Transmission Unit). The MTU of the network card inside the container should be less than or equal to the MTU of the host network card. For issues related to MTU and Docker, please refer to [1].

First, let me introduce my operating environment. I have launched two VMs on GCP, each with a specification of (GPU L4 * 8), and layered a container on top of this. The reason for using a container is that Nvidia provides a very convenient environment [2], which saves us the trouble of installing CUDA. One of the prerequisites for running Deepspeed on multiple nodes is to configure SSH for passwordless login. For relevant articles, please refer to [3][4].

My environment looks like this: docker_gwbridge is something automatically generated by Docker Swarm. For reference, please see [5].

  [main.py]                                                              [main.py]
 [nv-container]-------eth1:mtu1450--|                    |--eth1:mtu1450-[nv-container]
             docker_gwbridge:mtu1500|                    |   docker_gwbridge:mtu1500
[gcp vm        ]--------------------+------ens7:mtu1460--+--------------[gcp vm                ]

Finally, I changed ens7:mtu1460 to ens7:mtu1500 (setting the MTU to 1500), and successfully resolved the issue.

(Note 1) According to the official statement, it's best not to change it directly as it might cause problems [6]. So, what I actually did was create a new VPC network and then reconstruct the entire environment.
(Note 2) The reason I suspected the MTU was that when I was troubleshooting the deepspeed.initialize() hanging issue, I discovered something[7].

host1~$:ls       (sccuess)
host1~$:printenv (sccuess)
host2~$:ls       (sccuess)
host2~$:printenv (sccuess)
host1 -ssh-> host2~$: ls       (sccuess)
host1 -ssh-> host2~$: printenv (hangs on)
host1 -ssh-> host2~$: vim a.txt(hangs on)

[1]https://www.civo.com/learn/fixing-networking-for-docker
[2]https://hub.docker.com/r/rocker/cuda/tags
[3]https://github.com/microsoft/DeepSpeed/issues/4387#issuecomment-1823740873
[4]https://docs.docker.com/network/network-tutorial-overlay/#use-an-overlay-network-for-standalone-containers
[5]https://stackoverflow.com/questions/66445025/meaning-of-abbreviation-docker-gwbridge-or-what-does-gw-actually-mean
[6]https://cloud.google.com/vpc/docs/change-mtu-vpc-network
[7]https://unix.stackexchange.com/questions/14629/running-vim-on-a-remote-linux-machine-freezes-os-x-ssh-connection

The issue documents a multi-node DeepSpeed setup using main.py, Docker containers, GCP VMs, docker_gwbridge, and ens7 MTU settings. Start by reviewing the reported deepspeed.initialize() hang and the linked networking references. Done would require an agreed documentation or diagnostic change; the issue currently requests no specific modification.