swift 在使用vllm colocate模式的情况下进行GRPO训练 无报错卡死

Checklist / 检查清单

• I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。

Bug Description / Bug 描述

这个每个节点最后的log 这是卡主时每个节点的显存状况

How to Reproduce / 如何复现

运行脚本
#!/bin/bash

NCCL配置

export NCCL_DEBUG=${NCCL_DEBUG:-INFO}
export NCCL_SOCKET_IFNAME=eth0
export NCCL_IB_HCA=mlx5_2,mlx5_3,mlx5_6,mlx5_7
export NCCL_IB_DISABLE=0

time=$(date +"%Y%m%d_%H%M%S")
CONTAINER_NAME=${HOSTNAME}
exec > /work/model_train/qwen35-27B-sapo-node-4-exp-2/log/train_log/${time}-${CONTAINER_NAME}.log 2>&1

pip install weave
pip install "mcore-bridge==1.1.0" -U

pip install "vllm==0.19.0" -U
pip install "transformers==5.3.0" -U

echo "==================ms-swift================"
cd /work/ms-swift-4.1.1
pip install --upgrade pip
pip install -e .

cd /work/ms-swift-4.1.1
export MEGATRON_LM_PATH='/work/deploy_train/Megatron-LM-core_v0.16.1'

使用本地缓存避免共享盘冲突（每个节点独立缓存）

export HF_HOME=/tmp/hf_cache
export DATASETS_CACHE=/tmp/datasets_cache
export MODELSCOPE_CACHE=/tmp/modelscope_cache
export HUGGINGFACE_HUB_CACHE=/tmp/hf_cache
export HF_DATASETS_CACHE=/tmp/hf_datasets
export TRANSFORMERS_CACHE=/tmp/hf_models
export NCCL_TIMEOUT=1800
export ENCODE_TIMEOUT=300

创建本地缓存目录

mkdir -p $HF_HOME/datasets $DATASETS_CACHE $MODELSCOPE_CACHE $HUGGINGFACE_HUB_CACHE $HF_DATASETS_CACHE $TRANSFORMERS_CACHE

OUTPUT_DIR=/work/ms-swift-4.1.1/megatron_output/Qwen3.5-27B-sapo-node4-exp-2
mkdir -p $OUTPUT_DIR

================== 自动查找最新checkpoint ==================

在 OUTPUT_DIR 下所有 v* 子目录中搜索 checkpoint-*，找到编号最大的作为恢复点

LATEST_CKPT=""
LATEST_STEP=0

for vdir in "${OUTPUT_DIR}"/v*/; do
for ckpt_dir in "${vdir}"checkpoint-*/; do
if [ -d "$ckpt_dir" ]; then
step=$(basename "$ckpt_dir" | sed 's/checkpoint-//')
if [ "$step" -gt "$LATEST_STEP" ] 2>/dev/null; then
LATEST_STEP=$step
LATEST_CKPT=${ckpt_dir%/}
fi
fi
done
done

if [ -n "$LATEST_CKPT" ]; then
echo "==================Resuming from checkpoint================"
echo "Found latest checkpoint: ${LATEST_CKPT} (step ${LATEST_STEP})"
else
echo "==================No checkpoint found, starting from scratch================"
fi

================== END 自动查找 ==================

PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True'
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
FPS_MAX_FRAMES=12
IMAGE_MAX_TOKEN_NUM=512
VIDEO_MAX_TOKEN_NUM=512
NNODES=$1
NODE_RANK=$2
MASTER_ADDR=$3
MASTER_PORT=$4
NPROC_PER_NODE=8
SKIP_MULTIMODAL_MTP_VALIDATION=1
SWIFT_USE_MCORE_GDN=1
megatron rlhf
--rlhf_type grpo
--loss_type sapo
--tau_pos 1.0
--tau_neg 1.05
--importance_sampling_level token
--model /mnt/tenant-home_speed/muiltModel/Qwen/Qwen3.5-27B
--output_dir $OUTPUT_DIR
--save_safetensors true
--context_parallel_size 1
--tensor_model_parallel_size 8
--pipeline_model_parallel_size 1
--sequence_parallel true
--model_type qwen3_5
--template qwen3_5
--dynamic_sample true
--external_plugins /work/ms-swift-4.1.1/examples/train/plugins/multi_turn_reward.py
--reward_funcs multi_turn_accuracy multi_turn_format soft_overlong multi_turn_chinese_thinking
--dataset /work/ms-swift-4.1.1/examples/custom/data/muti_turn/converted_no_thought_bbox_full.jsonl
--soft_max_length 500
--soft_cache_length 200
--num_train_epochs 1
--global_batch_size 256
--micro_batch_size 4
--steps_per_generation 1
--num_generations 8
--use_vllm true
--vllm_mode colocate
--vllm_gpu_memory_utilization 0.4
--vllm_tensor_parallel_size 8
--vllm_mm_processor_cache_gb 0
--sleep_level 2
--vllm_max_model_len 8192
--max_length 8192
--max_completion_length 512
--tuner_type full
--torch_dtype bfloat16
--lr 1e-6
--lr_warmup_fraction 0.1
--min_lr 2e-7
--lr_decay_style cosine
--bf16 true
--beta 0.04
--epsilon 0.2
--epsilon_high 0.28
--overlong_filter true
--truncation_strategy delete
--offload_model false
--offload_bridge false
--offload_optimizer true
--logging_steps 1
--recompute_granularity full
--recompute_method uniform
--recompute_num_layers 16
--finetune
--dataloader_num_workers 8
--dataset_num_proc 8
--no_save_optim false
--no_save_rng
--save_steps 50
--save_total_limit 3
--attention_backend flash
--temperature 0.7
--top_k 50
--top_p 0.7
--repetition_penalty 1.2
--padding_free false
--log_completions true
--log_entropy true
--log_rollout_offpolicy_metrics true
--rollout_importance_sampling_mode token_mask
--rollout_importance_sampling_threshold 2.0
--optimizer_cpu_offload true
--use_precision_aware_optimizer true
--optimizer_offload_fraction 1
--overlap_grad_reduce false
--overlap_param_gather false
--report_to tensorboard

我使用4个节点强化qwen 3.5 27B
vllm==0.19.0
swift==4.1.1
transformers==5.3.0
megatron-core==0.16.1

Additional Information / 补充信息

No response

Start by running the provided four-node command with the listed ms-swift, vLLM, Transformers, and Megatron versions, then inspect the per-node logs and GPU state at the hang. Check the external plugin at examples/train/plugins/multi_turn_reward.py and the referenced dataset path while narrowing the failure to colocate GRPO execution. Done means reproducing the hang and identifying a verified cause or correction path.