训练时没有响应

所用的脚本：

#! /bin/bash
export CUDA_VISIBLE_DEVICES=4,5,6,7
GPUS_PER_NODE=4

NNODES=1
MASTER_ADDR="localhost"
MASTER_PORT=12346

OPTS=""
OPTS+=" --use-delta"
OPTS+=" --model-config /home/zyz/.cache/modelscope/hub/OpenBMB/cpm-bee-10b/config/cpm-bee-10b.json"
OPTS+=" --dataset /home/zyz/cpm/datasets/dataset.json"
OPTS+=" --eval_dataset /home/zyz/cpm/datasets/step/data"
OPTS+=" --epoch 100"
OPTS+=" --batch-size 5"
OPTS+=" --train-iters 100"
OPTS+=" --save-name cpm_bee_finetune"
OPTS+=" --max-length 2048"
OPTS+=" --save results/"
OPTS+=" --lr 0.0001"
OPTS+=" --inspect-iters 100"
OPTS+=" --warmup-iters 1"
OPTS+=" --eval-interval 1000"
OPTS+=" --early-stop-patience 5"
OPTS+=" --lr-decay-style noam"
OPTS+=" --weight-decay 0.01"
OPTS+=" --clip-grad 1.0"
OPTS+=" --loss-scale 32768"
OPTS+=" --start-step 0"
OPTS+=" --load /home/zyz/.cache/modelscope/hub/OpenBMB/cpm-bee-10b/pytorch_model.bin"

CMD="torchrun --nnodes=${NNODES} --nproc_per_node=${GPUS_PER_NODE} --rdzv_id=1 --rdzv_backend=c10d --rdzv_endpoint=${MASTER_ADDR}:${MASTER_PORT} finetune_cpm_bee.py ${OPTS}"

echo ${CMD}
$CMD

```
`
输出结果：`

```
> nohup: 忽略输入
> torchrun --nnodes=1 --nproc_per_node=4 --rdzv_id=1 --rdzv_backend=c10d --rdzv_endpoint=localhost:12346 finetune_cpm_bee.py --use-delta --model-config /home/zyz/.cache/modelscope/hub/OpenBMB/cpm-bee-10b/config/cpm-bee-10b.json --dataset /home/zyz/cpm/datasets/dataset.json --eval_dataset /home/zyz/cpm/datasets/step/data --epoch 100 --batch-size 5 --train-iters 100 --save-name cpm_bee_finetune --max-length 2048 --save results/ --lr 0.0001 --inspect-iters 100 --warmup-iters 1 --eval-interval 1000 --early-stop-patience 5 --lr-decay-style noam --weight-decay 0.01 --clip-grad 1.0 --loss-scale 32768 --start-step 0 --load /home/zyz/.cache/modelscope/hub/OpenBMB/cpm-bee-10b/pytorch_model.bin
> [2024-03-28 20:20:35,784] torch.distributed.run: [WARNING] 
> [2024-03-28 20:20:35,784] torch.distributed.run: [WARNING] *****************************************
> [2024-03-28 20:20:35,784] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
> [2024-03-28 20:20:35,784] torch.distributed.run: [WARNING] *****************************************
> /home/zyz/anaconda3/envs/cpm/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
>   warnings.warn(
> /home/zyz/anaconda3/envs/cpm/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
>   warnings.warn(
> /home/zyz/anaconda3/envs/cpm/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
>   warnings.warn(
> /home/zyz/anaconda3/envs/cpm/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
>   warnings.warn(
> ====================== Initialization ======================
> rank :          0
> local_rank :    0
> world_size :    4
> local_size :    4
> master :        aa-ESC8000-G4:53279
> device :        0
> cpus :          [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1
>                 3, 14, 15, 16, 17, 18, 19]
> 
> /home/zyz/anaconda3/envs/cpm/lib/python3.10/site-packages/bmtrain/synchronize.py:14: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.)
>   barrier = torch.cuda.FloatTensor([1])
> /home/zyz/anaconda3/envs/cpm/lib/python3.10/site-packages/bmtrain/synchronize.py:15: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
>   nccl.allReduce(barrier.storage(), barrier.storage(), 'sum', config['comm'])
> /home/zyz/anaconda3/envs/cpm/lib/python3.10/site-packages/bmtrain/synchronize.py:14: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.)
>   barrier = torch.cuda.FloatTensor([1])
> /home/zyz/anaconda3/envs/cpm/lib/python3.10/site-packages/bmtrain/synchronize.py:15: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
>   nccl.allReduce(barrier.storage(), barrier.storage(), 'sum', config['comm'])
> /home/zyz/anaconda3/envs/cpm/lib/python3.10/site-packages/bmtrain/synchronize.py:14: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.)
>   barrier = torch.cuda.FloatTensor([1])
> /home/zyz/anaconda3/envs/cpm/lib/python3.10/site-packages/bmtrain/synchronize.py:15: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
>   nccl.allReduce(barrier.storage(), barrier.storage(), 'sum', config['comm'])
> /home/zyz/anaconda3/envs/cpm/lib/python3.10/site-packages/bmtrain/synchronize.py:14: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.)
>   barrier = torch.cuda.FloatTensor([1])
> /home/zyz/anaconda3/envs/cpm/lib/python3.10/site-packages/bmtrain/synchronize.py:15: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
>   nccl.allReduce(barrier.storage(), barrier.storage(), 'sum', config['comm'])
```

`在该界面一直停留，没有响应

Start by reproducing the reported torchrun command with finetune_cpm_bee.py and inspect the initialization output around bmtrain/synchronize.py. Compare the behavior and logs from all four ranks; done means identifying why execution remains at initialization and documenting or fixing the cause.