Selecting a range of ports to try until an open port is found

Is your feature request related to a problem? Please describe.

It is related to #191. Specifically my comment there. I have opened a new issue as requested to discuss the potential of a new feature that might help to prevent the original issue from happening.

The issue occurs that the host-address is already in use by another user or another process. For instance a node with 4 GPUs but two separate jobs with deepspeed that both use 2 GPUs. So their hostfile is identical. This triggers a "port already in use".

Describe the solution you'd like

I thought that it might be useful to add the option to specify a range of ports (e.g. 16000:25000) as an environment variable or as a CLI option. DeepSpeed could then retry connecting through that range of ports until a suitable port is found, rather than just exiting after the default port is already taken.

@tjruwase mentions that this might be slow. I do not think that that is a huge issue because the search only has to happen once (at the start of the process) until an open port is found and it does not happen in the current, default, scenario. So the user should be aware that when they specify a range of ports, that time until connection may take longer. Through socket in Python, I think it should be possible to write this for many different platforms without relying on e.g. netstat, but I am not 100% sure how feasible it is in the larger scope of DeepSpeed.

Start by tracing DeepSpeed's current default-port connection path and the Python socket handling used during distributed startup. Review how a CLI option or environment variable could express a port range and how retries should behave when the host address is already in use. Done means a specified range is tried until an available port is found, while the default behavior remains unchanged.