8卡训练 Qwen3-32B-VL-Instruct 会卡住

Checklist / 检查清单

• I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。

Bug Description / Bug 描述

--deepspeed zero3

py-spy dump --pid 1521571 结果：

Python v3.10.13 (/tmp/envs/py310/bin/python3.10)

Thread 1521571 (active): "MainThread"
    _allgather_params (deepspeed/runtime/zero/partition_parameters.py:1948)
    _all_gather (deepspeed/runtime/zero/partition_parameters.py:1499)
    wrapped_fn (deepspeed/utils/nvtx.py:15)
    all_gather (deepspeed/runtime/zero/partition_parameters.py:1143)
    __enter__ (deepspeed/runtime/zero/partition_parameters.py:2224)
    get_n_params_grads (swift/utils/torch_utils.py:45)
    get_model_parameter_info (swift/utils/torch_utils.py:52)
    run (swift/llm/train/sft.py:192)
    wrapper (swift/ray/base.py:170)
    main (swift/llm/base.py:49)
    sft_main (swift/llm/train/sft.py:353)
    <module> (swift/cli/sft.py:20)
Thread 1522405 (idle): "Thread-1"
    wait (threading.py:324)
    wait (threading.py:607)
    run (tqdm/_monitor.py:60)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)

How to Reproduce / 如何复现

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
NPROC_PER_NODE=8 \
nohup swift sft \
--model /tmp/Qwen3-VL-32B-Instruct/ \
--dataset  \
train.jsonl \
--val_dataset eval_200.jsonl  \
--train_type lora \
--torch_dtype bfloat16 \
--num_train_epochs 1 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--learning_rate 2e-5 \
--lora_rank 300 \
--lora_alpha 300 \
--freeze_llm false \
--freeze_vit true \
--gradient_accumulation_steps 1 \
--save_steps 500 \
--save_total_limit 5 \
--logging_steps 5 \
--max_new_tokens 12800 \
--max_length 16000 \
--output_dir outputs/Qwen3-VL-32B-Instruct/ \
--resume_from_checkpoint outputs/Qwen3-VL-32B-Instruct/v0-20260219-091207/checkpoint-22966/ \
--resume_only_model true \
--ignore_data_skip true \
--warmup_steps 300 \
--dataloader_num_workers 0 \
--optim adamw_8bit \
--attn_impl flash_attn \
--do_eval false \
--truncation_strategy delete \
--torch_empty_cache_steps 20 \
--max_pixels 409600 --deepspeed zero3 > ./train.log 2>&1 &

Additional Information / 补充信息

日志信息：

[notebook-e54a54f43f75-worker-0][1521576:1522375][0][state_monitor.cc:250] NCCL WARN StateMonitor:3196561941ad3373:Rank6 : no pending work(s) completed over 60.00 seconds, channel 0, enqueued 3495, completed 3438

[notebook-e54a54f43f75-worker-0][1521577:1522384][0][state_monitor.cc:250] NCCL WARN StateMonitor:3196561941ad3373:Rank7 : no pending work(s) completed over 60.00 seconds, channel 0, enqueued 3495, completed 3438

[notebook-e54a54f43f75-worker-0][1521571:1522377][0][state_monitor.cc:250] NCCL WARN StateMonitor:3196561941ad3373:Rank1 : no pending work(s) completed over 60.00 seconds, channel 0, enqueued 3495, completed 3438

[notebook-e54a54f43f75-worker-0][1521570:1522374][0][state_monitor.cc:250] NCCL WARN StateMonitor:3196561941ad3373:Rank0 : no pending work(s) completed over 60.00 seconds, channel 0, enqueued 3495, completed 3438

[notebook-e54a54f43f75-worker-0][1521572:1522376][0][state_monitor.cc:250] NCCL WARN StateMonitor:3196561941ad3373:Rank2 : no pending work(s) completed over 60.00 seconds, channel 0, enqueued 3495, completed 3438

[notebook-e54a54f43f75-worker-0][1521573:1522385][0][state_monitor.cc:250] NCCL WARN StateMonitor:3196561941ad3373:Rank3 : no pending work(s) completed over 60.00 seconds, channel 0, enqueued 3495, completed 3438

[notebook-e54a54f43f75-worker-0][1521574:1522383][0][state_monitor.cc:250] NCCL WARN StateMonitor:3196561941ad3373:Rank4 : no pending work(s) completed over 60.00 seconds, channel 0, enqueued 3495, completed 3438

[notebook-e54a54f43f75-worker-0][1521571:1522382][0][state_monitor.cc:108] NCCL WARN StateMonitor:3196561941ad3373:Rank1 : trigger debugging dump

[notebook-e54a54f43f75-worker-0][1521576:1522380][0][state_monitor.cc:108] NCCL WARN StateMonitor:3196561941ad3373:Rank6 : trigger debugging dump

[notebook-e54a54f43f75-worker-0][1521572:1522381][0][state_monitor.cc:108] NCCL WARN StateMonitor:3196561941ad3373:Rank2 : trigger debugging dump

[notebook-e54a54f43f75-worker-0][1521574:1522389][0][state_monitor.cc:108] NCCL WARN StateMonitor:3196561941ad3373:Rank4 : trigger debugging dump

[notebook-e54a54f43f75-worker-0][1521573:1522387][0][state_monitor.cc:108] NCCL WARN StateMonitor:3196561941ad3373:Rank3 : trigger debugging dump

[notebook-e54a54f43f75-worker-0][1521577:1522386][0][state_monitor.cc:108] NCCL WARN StateMonitor:3196561941ad3373:Rank7 : trigger debugging dump

[notebook-e54a54f43f75-worker-0][1521570:1522379][0][state_monitor.cc:108] NCCL WARN StateMonitor:3196561941ad3373:Rank0 : trigger debugging dump

[notebook-e54a54f43f75-worker-0][1521575:1522388][0][state_monitor.cc:108] NCCL WARN StateMonitor:3196561941ad3373:Rank5 : trigger debugging dump

Start with swift/utils/torch_utils.py at get_n_params_grads and get_model_parameter_info, then follow the call from swift/llm/train/sft.py:192. Reproduce the supplied eight-GPU swift sft command with DeepSpeed ZeRO-3 and inspect the py-spy and NCCL warnings. Done means the training no longer hangs during parameter gathering and the reported configuration can progress.