[BUG] Cannot get elastic scaling to work

Cannot elastically scale up the number of workers
When I want to scale up the number of workers from 2 to 3, the training crashes.

To Reproduce
Steps to reproduce the behavior:

1. Use https://github.com/microsoft/DeepSpeedExamples/blob/arpan/elastic_scripts/elastic_training/train_bert_ds_el_script.py
2. Run

deepspeed --num_nodes 2 --max_num_nodes 4 --min_num_nodes 1 \
          --num_gpus 1 --force_multi --hostfile hosts_el \
          --elastic_training --master_port 49091 --master_addr worker-0  \
          train_bert_ds_el_script.py --checkpoint_dir ./checks_el

3. Run the same command with hostfile that has 3 workers on worker with rank 3.
4. See error

Expected behavior
Continue training with 3 workers.

Error

deepspeed-worker-2-1  | worker-0: [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:49091 (errno: 98 - Address already in use).
deepspeed-worker-2-1  | worker-0: [W socket.cpp:426] [c10d] The server socket has failed to bind to ?UNKNOWN? (errno: 98 - Address already in use).
deepspeed-worker-2-1  | worker-0: [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address.

deepspeed-worker-0-1  | worker-0: Sending process 175 closing signal SIGTERM
deepspeed-worker-0-1  | worker-1: Sending process 100 closing signal SIGTERM
deepspeed-worker-2-1  | worker-1: => set cuda device = 0 Global Rank = 3
deepspeed-worker-0-1  | worker-1: => set cuda device = 0 Global Rank = 2
deepspeed-worker-2-1  | worker-0: => set cuda device = 0 Global Rank = 1
deepspeed-worker-0-1  | worker-0: => set cuda device = 0 Global Rank = 0
deepspeed-worker-2-1  | worker-2: => set cuda device = 0 Global Rank = 4

deepspeed-worker-2-1  | worker-0: Duplicate GPU detected : rank 1 and rank 0 both on CUDA device 1000

System info (please complete the following information):

• OS: Ubuntu 22.04
• 1 A6000 GPU per worker
• Interconnects Ethernet
• Python version 3.10

Launcher context
deepspeed with --elastic_training

Docker context
Image based on nvidia/cuda:11.7.1-cudnn8-devel-ubuntu22.04
With Deepspeed from https://github.com/microsoft/DeepSpeed/tree/arpan/elasticity

Assumption
The new worker is not added to the training but an now training with 3 workers is started.