qwen3-vl 在全参sft后的模型上做grpo，训练一定step后遇到报错Watchdog caught collective operation timeout

Describe the bug
qwen3-vl 在全参sft后的模型上做grpo，训练一定step后遇到报错Watchdog caught collective operation timeout，报错如下（我的训练数据在本地，图片用的本地路径）：

Train: 62%|██████▏ | 5559/9008 [9:33:11<5:18:51, 5.55s/it]
Train: 62%|██████▏ | 5559/9008 [9:33:11<5:18:51, 5.55s/it]
0%| | 0/48 [00:00<?, ?it/s]
2%|▏ | 1/48 [00:00<00:14, 3.29it/s]
0%| | 0/48 [00:00<?, ?it/s]
100%|██████████| 48/48 [00:00<00:00, 103.13it/s]
[rank0]:[E1206 21:12:01.852621435 ProcessGroupNCCL.cpp:685] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=434159, OpType=ALLREDUCE, NumelIn=2, NumelOut=2, Timeout(ms)=600000) ran for 600013 milliseconds before timing out.
[rank0]:[E1206 21:12:01.858423203 ProcessGroupNCCL.cpp:2252] [PG ID 5 PG GUID 15 Rank 0] failure detected by watchdog at work sequence id: 434159 PG status: last enqueued work: 434159, last completed work: 434158
[rank0]:[E1206 21:12:01.858446367 ProcessGroupNCCL.cpp:732] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank0]:[E1206 21:12:01.858509006 ProcessGroupNCCL.cpp:2584] [PG ID 5 PG GUID 15 Rank 0] First PG on this rank to signal dumping.
[rank0]:[E1206 21:12:01.993362399 ProcessGroupNCCL.cpp:1870] [PG ID 0 PG GUID 0 Rank 0] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: -1, last completed NCCL work: -1.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc.
[rank1]:[E1206 21:12:01.993454221 ProcessGroupNCCL.cpp:1806] [PG ID 0 PG GUID 0 Rank 1] Observed flight recorder dump signal from another rank via TCPStore.
[rank1]:[E1206 21:12:01.993642221 ProcessGroupNCCL.cpp:1870] [PG ID 0 PG GUID 0 Rank 1] Received a dump signal due to a collective timeout from rank 0 and we will try our best to dump the debug info. Last enqueued NCCL work: -1, last completed NCCL work: -1.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc.
[rank1]:[E1206 21:12:01.994663779 ProcessGroupNCCL.cpp:1589] [PG ID 0 PG GUID 0 Rank 1] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank0]:[E1206 21:12:01.994691693 ProcessGroupNCCL.cpp:1589] [PG ID 0 PG GUID 0 Rank 0] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank0]:[E1206 21:13:01.858621685 ProcessGroupNCCL.cpp:746] [Rank 0] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank0]:[E1206 21:13:01.858686189 ProcessGroupNCCL.cpp:760] [Rank 0] To avoid data inconsistency, we are taking the entire process down.
[rank0]:[E1206 21:13:01.861998981 ProcessGroupNCCL.cpp:2068] [PG ID 5 PG GUID 15 Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=434159, OpType=ALLREDUCE, NumelIn=2, NumelOut=2, Timeout(ms)=600000) ran for 600013 milliseconds before timing out.
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:688 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x7f2b8e368eb0 in /home/tiger/.local/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x247 (0x7f2b31493147 in /home/tiger/.local/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x1591 (0x7f2b31496b61 in /home/tiger/.local/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::Watchdog::run() + 0xd2 (0x7f2b31497ec2 in /home/tiger/.local/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0xd44a3 (0x7f2bc81844a3 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #5: + 0x89144 (0x7f2bcc51a144 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #6: + 0x1097dc (0x7f2bcc59a7dc in /usr/lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 5 PG GUID 15 Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=434159, OpType=ALLREDUCE, NumelIn=2, NumelOut=2, Timeout(ms)=600000) ran for 600013 milliseconds before timing out.
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:688 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x7f2b8e368eb0 in /home/tiger/.local/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x247 (0x7f2b31493147 in /home/tiger/.local/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x1591 (0x7f2b31496b61 in /home/tiger/.local/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::Watchdog::run() + 0xd2 (0x7f2b31497ec2 in /home/tiger/.local/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0xd44a3 (0x7f2bc81844a3 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #5: + 0x89144 (0x7f2bcc51a144 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #6: + 0x1097dc (0x7f2bcc59a7dc in /usr/lib/x86_64-linux-gnu/libc.so.6)

Exception raised from run at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:2074 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x7f2b8e368eb0 in /home/tiger/.local/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: + 0xe1c1a1 (0x7f2b3146f1a1 in /home/tiger/.local/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0x9468e6 (0x7f2b30f998e6 in /home/tiger/.local/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: + 0xd44a3 (0x7f2bc81844a3 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #4: + 0x89144 (0x7f2bcc51a144 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #5: + 0x1097dc (0x7f2bcc59a7dc in /usr/lib/x86_64-linux-gnu/libc.so.6)

[rank1]:[F1206 21:20:01.996889274 ProcessGroupNCCL.cpp:1614] [PG ID 0 PG GUID 0 Rank 1] [PG ID 0 PG GUID 0 Rank 1] Terminating the process after attempting to dump debug info, due to collective timeout or exception.

我使用的gpu6和7作为vllm推理卡，可以看到在出现这个问题之前，这两张卡的显存占用一直都不高，并且运行正常

Your hardware and system info
Write your system info like CUDA version/system/GPU/torch version here(在这里给出硬件信息和系统信息，如CUDA版本，系统，GPU型号和torch版本等)
库的版本：
transformers==4.57.1
qwen_vl_utils==0.0.14
ms-swift==3.10.3
deepspeed==0.17.6
vllm==0.11.0
trl==0.24.0
torch==2.8.0

我使用的训练脚本是，用的8卡H100：
#!/bin/bash

8 * 80GiB 全参grpo训练

Install dependencies

pip install -r requirements.txt
pip install flash-attn --no-build-isolation

Set W&B project and run name

打印一下环境信息，方便排查

echo "Starting training with WandB Project: $WANDB_PROJECT, Name: $WANDB_NAME"

--- 自动寻找一个空闲端口给 vLLM 使用 ---

get_free_port() {
python3 -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()'
}

生成 vLLM 的 HTTP 服务端口 (专门给 vLLM 的 HTTP API 使用（swift rlhf 会通过这个端口发请求）)

export PORT=$(get_free_port)
echo "Auto-assigned vLLM Port: $PORT"

新增】生成 vLLM 内部分布式通信端口 (专门给 vLLM 内部的 2 张卡（GPU 6,7）做分布式通信使用)

必须确保这个端口和 PORT 不同，也和全局 MASTER_PORT 不同

export VLLM_DIST_PORT=$(python3 -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()')
echo "Auto-assigned vLLM Internal Distributed Port: $VLLM_DIST_PORT"

PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True'
MASTER_ADDR=127.0.0.1
MASTER_PORT=$VLLM_DIST_PORT
IMAGE_MAX_TOKEN_NUM=1024
CUDA_VISIBLE_DEVICES=6,7
swift rollout
--model /xx/checkpoint-1500
--port $PORT
--host 127.0.0.1
--vllm_tensor_parallel_size 1
--vllm_data_parallel_size 2 &
echo "Waiting for vLLM to start..."
sleep 120
echo "Start rlhf..."

PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True'
MASTER_ADDR=$MASTER_ADDR
MASTER_PORT=$(echo $ARNOLD_WORKER_0_PORT | cut -d ',' -f 1)
IMAGE_MAX_TOKEN_NUM=1024
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5
NPROC_PER_NODE=6
swift rlhf
--rlhf_type grpo
--model /xx/checkpoint-1500
--dataset /xx/grpo_output_fix_prompt_img224.jsonl
--val_dataset /xx/grpo_output_fix_prompt_img224_sampled_5_percent.jsonl
--load_from_cache_file true
--split_dataset_ratio 0.0
--external_plugins swift_customization/plugin.py
--reward_funcs xx
--use_vllm true
--vllm_mode server
--vllm_server_host 127.0.0.1
--vllm_server_port $PORT
--train_type full
--torch_dtype bfloat16
--max_length 1024
--truncation_strategy delete
--max_completion_length 128
--num_train_epochs 1
--per_device_train_batch_size 16
--per_device_eval_batch_size 16
--learning_rate 2e-6
--gradient_accumulation_steps 1
--gradient_checkpointing true
--save_strategy 'steps'
--eval_strategy 'steps'
--eval_steps 400
--save_steps 400
--logging_steps 1
--output_dir /xx/output_grpo
--warmup_ratio 0.05
--dataloader_num_workers 8
--num_generations 8
--temperature 1.1
--top_p 0.8
--epsilon_high 0.28
--dynamic_sample true
--max_resample_times 3
--importance_sampling_level sequence
--deepspeed zero3
--report_to wandb
--run_name "$WANDB_NAME"
--log_completions true
--log_entropy false
--async_generate false
--beta 0.01
--attn_impl flash_attn
--padding_free true

Start with the supplied training script, especially the separate vLLM and swift rlhf launch commands, and inspect the NCCL ALLREDUCE timeout after step 5559. Compare the MASTER_PORT, VLLM_DIST_PORT, CUDA_VISIBLE_DEVICES, and distributed settings used by both processes. Done means identifying a reproducible cause and confirming a configuration or code change allows training to continue without the collective timeout.