[BUG] Multi-node fine-tuning with thunderbolt

Describe the bug

I want to perform multi-node fine-tuning using deepspeed and Llama Factory. I offload my model to cpu-cpu stage3. Using LAN and Thunderbolt for SSH connections, the connection functions work normally. When I connected using LAN, the fine-tuning works normally, but when I switch to Thunderbolt, it gets stuck. Unfortunately, there are no error messages in the program. Has anyone encountered the same issue?

To Reproduce

package version:

• torch=2.2.2+cu121
• torchaudio=2.2.2+cu121
• torchvision=0.17.2+cu121
• deepspeed=0.13.4
• transformers=4.39.3

system info:

• system 1:
  • MB : gigabyte z790 XTREME
  • GPU: RTX 4090
• system 2:
  • MB : gigabyte z790 XTREME
  • GPU: RTX 4080

training script:

$gpus=1

$train_batch=1

deepspeed --hostfile=../../hostfile.txt \
    --num_gpus $gpus ../../src/train_bash.py \
    --deepspeed ds_z3_offload_cpu_cpu_config.json \
    --stage sft \
    --do_train \
    --model_name_or_path meta-llama/Llama-2-7b-chat-hf \
    --dataset alpaca_gpt4_en \
    --dataset_dir ../../data \
    --template default \
    --finetuning_type lora \
    --output_dir ../../saves/llama3_lorafull/sft \
    --overwrite_cache \
    --overwrite_output_dir \
    --cutoff_len $seq_len \
    --lr_scheduler_type cosine \
    --logging_steps 10 \
    --per_device_train_batch_size $train_batch \
    --per_device_eval_batch_size 1 \
    --gradient_accumulation_steps 1 \
    --eval_steps 100 \
    --evaluation_strategy steps \
    --learning_rate 5e-5 \
    --num_train_epochs 1.0 \
    --lora_target q_proj,v_proj \
    --lora_r 16 \
    --lora_alpha 32 \
    --lora_dropout 0.05 \
    --max_samples 3000 \
    --val_size 0.1 \
    --plot_loss \
    --bf16 2>&1 | tee $filename
    

hostfile:

192.168.1.1 slots=1
192.168.1.2 slots=1

offload config:

{
  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": "auto",
  "gradient_accumulation_steps": "auto",
  "gradient_clipping": "auto",
  "fp16": {
    "enabled": "auto",
    "loss_scale": 0,
    "loss_scale_window": 1000,
    "initial_scale_power": 16,
    "hysteresis": 2,
    "min_loss_scale": 1
  },

  
   "optimizer": {
        "type": "AdamW",
        "params": {
            "lr": "auto",
            "betas": "auto",
            "eps": "auto",
            "weight_decay": "auto"
        }
    },

    "scheduler": {
        "type": "WarmupLR",
        "params": {
            "warmup_min_lr": "auto",
            "warmup_max_lr": "auto",
            "warmup_num_steps": "auto"
        }
    },
  
  "zero_optimization": {
    "stage": 3,
    "offload_optimizer": {
                "device": "cpu",
                "nvme_path": "/media/user/nvme0",
                "pin_memory": true,
                "ratio": 1,
                "buffer_count": 5,
                "fast_init": false
            },
    "offload_param": {
                "device": "cpu",
                "nvme_path": "/media/user/nvme0",
                "pin_memory": true,
                "buffer_count": 5,
                "buffer_size": 1e8,
                "max_in_cpu": 1e9
            },
            
    "round_robin_gradients": true,
    "overlap_comm": true,
    "contiguous_gradients": true,
    "sub_group_size": 1e9,
    "reduce_bucket_size": "auto",
    "stage3_prefetch_bucket_size": "auto",
    "stage3_param_persistence_threshold": "auto",
    "stage3_max_live_parameters": 1e9,
    "stage3_max_reuse_distance": 0,
    "stage3_gather_16bit_weights_on_model_save": true
  }
}

LOG:(The program is stuck on the last line.)

[2024-07-11 11:16:24,229] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-07-11 11:16:24,855] [INFO] [runner.py:463:main] Using IP address of 172.17.0.1 for node 192.168.1.1
[2024-07-11 11:16:24,856] [INFO] [multinode_runner.py:80:get_cmd] Running on the following workers: 192.168.1.1,192.168.1.2
[2024-07-11 11:16:24,856] [INFO] [runner.py:568:main] cmd = pdsh -S -f 1024 -w 192.168.1.1,192.168.1.2 export NCCL_P2P_DISABLE=1; export NCCL_SOCKET_IFNAME=thunderbolt0,thunderbolt0; export NCCL_DEBUG=DEBUG; export NCCL_IB_DISABLE=1; export PYTHONPATH=/home/trx50/Project/factory/LLaMA-Factory/examples/full_multi_gpu;  cd /home/trx50/Project/factory/LLaMA-Factory/examples/full_multi_gpu; /home/trx50/.virtualenvs/factory/bin/python -u -m deepspeed.launcher.launch --world_info=eyIxOTIuMTY4LjEuMSI6IFswXSwgIjE5Mi4xNjguMS4yIjogWzBdfQ== --node_rank=%n --master_addr=172.17.0.1 --master_port=29500 ../../src/train_bash.py --deepspeed ds_z3_offload_cpu_cpu_config.json --stage sft --do_train --model_name_or_path meta-llama/Llama-2-7b-chat-hf --dataset alpaca_gpt4_en --dataset_dir ../../data --template default --finetuning_type lora --output_dir ../../saves/llama3_lorafull/sft --overwrite_cache --overwrite_output_dir --cutoff_len 2048 --lr_scheduler_type cosine --logging_steps 10 --per_device_train_batch_size 1 --per_device_eval_batch_size 1 --gradient_accumulation_steps 1 --eval_steps 100 --evaluation_strategy steps --learning_rate 5e-5 --num_train_epochs 1.0 --lora_target q_proj,v_proj --lora_r 16 --lora_alpha 32 --lora_dropout 0.05 --max_samples 3000 --val_size 0.1 --plot_loss --bf16
192.168.1.1: [2024-07-11 11:16:25,980] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
192.168.1.2: [2024-05-18 06:57:44,090] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
192.168.1.1: [2024-07-11 11:16:26,210] [INFO] [launch.py:138:main] 0 NCCL_P2P_DISABLE=1
192.168.1.1: [2024-07-11 11:16:26,210] [INFO] [launch.py:138:main] 0 NCCL_SOCKET_IFNAME=thunderbolt0,thunderbolt0
192.168.1.1: [2024-07-11 11:16:26,210] [INFO] [launch.py:138:main] 0 NCCL_DEBUG=DEBUG
192.168.1.1: [2024-07-11 11:16:26,210] [INFO] [launch.py:138:main] 0 NCCL_IB_DISABLE=1
192.168.1.1: [2024-07-11 11:16:26,210] [INFO] [launch.py:145:main] WORLD INFO DICT: {'192.168.1.1': [0], '192.168.1.2': [0]}
192.168.1.1: [2024-07-11 11:16:26,210] [INFO] [launch.py:151:main] nnodes=2, num_local_procs=1, node_rank=0
192.168.1.1: [2024-07-11 11:16:26,210] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(<class 'list'>, {'192.168.1.1': [0], '192.168.1.2': [1]})
192.168.1.1: [2024-07-11 11:16:26,210] [INFO] [launch.py:163:main] dist_world_size=2
192.168.1.1: [2024-07-11 11:16:26,210] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=0
192.168.1.1: [2024-07-11 11:16:26,210] [INFO] [launch.py:253:main] process 22275 spawned with command: ['/home/trx50/.virtualenvs/factory/bin/python', '-u', '../../src/train_bash.py', '--local_rank=0', '--deepspeed', 'ds_z3_offload_cpu_cpu_config.json', '--stage', 'sft', '--do_train', '--model_name_or_path', 'meta-llama/Llama-2-7b-chat-hf', '--dataset', 'alpaca_gpt4_en', '--dataset_dir', '../../data', '--template', 'default', '--finetuning_type', 'lora', '--output_dir', '../../saves/llama3_lorafull/sft', '--overwrite_cache', '--overwrite_output_dir', '--cutoff_len', '2048', '--lr_scheduler_type', 'cosine', '--logging_steps', '10', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '1', '--gradient_accumulation_steps', '1', '--eval_steps', '100', '--evaluation_strategy', 'steps', '--learning_rate', '5e-5', '--num_train_epochs', '1.0', '--lora_target', 'q_proj,v_proj', '--lora_r', '16', '--lora_alpha', '32', '--lora_dropout', '0.05', '--max_samples', '3000', '--val_size', '0.1', '--plot_loss', '--bf16']
192.168.1.2: [2024-05-18 06:57:44,366] [INFO] [launch.py:138:main] 1 NCCL_P2P_DISABLE=1
192.168.1.2: [2024-05-18 06:57:44,366] [INFO] [launch.py:138:main] 1 NCCL_SOCKET_IFNAME=thunderbolt0,thunderbolt0
192.168.1.2: [2024-05-18 06:57:44,366] [INFO] [launch.py:138:main] 1 NCCL_DEBUG=DEBUG
192.168.1.2: [2024-05-18 06:57:44,366] [INFO] [launch.py:138:main] 1 NCCL_IB_DISABLE=1
192.168.1.2: [2024-05-18 06:57:44,366] [INFO] [launch.py:145:main] WORLD INFO DICT: {'192.168.1.1': [0], '192.168.1.2': [0]}
192.168.1.2: [2024-05-18 06:57:44,366] [INFO] [launch.py:151:main] nnodes=2, num_local_procs=1, node_rank=1
192.168.1.2: [2024-05-18 06:57:44,366] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(<class 'list'>, {'192.168.1.1': [0], '192.168.1.2': [1]})
192.168.1.2: [2024-05-18 06:57:44,366] [INFO] [launch.py:163:main] dist_world_size=2
192.168.1.2: [2024-05-18 06:57:44,366] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=0
192.168.1.2: [2024-05-18 06:57:44,366] [INFO] [launch.py:253:main] process 11393 spawned with command: ['/home/trx50/.virtualenvs/factory/bin/python', '-u', '../../src/train_bash.py', '--local_rank=0', '--deepspeed', 'ds_z3_offload_cpu_cpu_config.json', '--stage', 'sft', '--do_train', '--model_name_or_path', 'meta-llama/Llama-2-7b-chat-hf', '--dataset', 'alpaca_gpt4_en', '--dataset_dir', '../../data', '--template', 'default', '--finetuning_type', 'lora', '--output_dir', '../../saves/llama3_lorafull/sft', '--overwrite_cache', '--overwrite_output_dir', '--cutoff_len', '2048', '--lr_scheduler_type', 'cosine', '--logging_steps', '10', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '1', '--gradient_accumulation_steps', '1', '--eval_steps', '100', '--evaluation_strategy', 'steps', '--learning_rate', '5e-5', '--num_train_epochs', '1.0', '--lora_target', 'q_proj,v_proj', '--lora_r', '16', '--lora_alpha', '32', '--lora_dropout', '0.05', '--max_samples', '3000', '--val_size', '0.1', '--plot_loss', '--bf16']
192.168.1.2: [2024-05-18 06:57:45,536] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
192.168.1.1: [2024-07-11 11:16:27,773] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
192.168.1.1: [2024-07-11 11:16:28,240] [INFO] [comm.py:637:init_distributed] cdb=None
192.168.1.1: [2024-07-11 11:16:28,241] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
192.168.1.2: [2024-05-18 06:57:46,373] [INFO] [comm.py:637:init_distributed] cdb=None

Expected behavior

Execute the training normally and produce results.

Start with DeepSpeed's runner.py, multinode_runner.py, launch.py, and comm.py entry points, using the provided logs to trace initialization over thunderbolt0. Reproduce the multi-node command with Thunderbolt and LAN, compare the NCCL and rendezvous behavior, and identify why initialization stops without an error. Done means the supplied training command completes successfully over Thunderbolt and produces results.