qwen3-VL on policy distillation

Checklist / 检查清单

• I have searched existing issues, and this is a new question or discussion topic. / 我已经搜索过现有的 issues，确认这是一个新的问题与讨论。

Question Description / 问题描述

作者您好！我对qwen3-vl使用OPD训练，配置如下

CUDA_VISIBLE_DEVICES=9,10,11,12 \
swift rollout \
    --model /r-contentsecurity/share/checkpoints/opensources/Qwen3-VL-8B-Instruct \
    --vllm_max_model_len 24192 \
    --host 0.0.0.0 \
    --port 18924 \
    --vllm_tensor_parallel_size 2 \
    --vllm_data_parallel_size 2 \

# On-Policy Distillation https://thinkingmachines.ai/blog/on-policy-distillation/



NPROC_PER_NODE=8 \
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
CUDA_VISIBLE_DEVICES=1,2,3,4,5,6,7,8 \
swift rlhf \
    --rlhf_type gkd \
    --model /r-contentsecurity/share/checkpoints/opensources/Qwen3-VL-8B-Instruct \
    --teacher_model /r-contentsecurity/share/checkpoints/opensources/Qwen3-VL-32B-Instruct \
    --tuner_type full \
    --dataset /mnt/nas/yanlong/code/new_benchmark/dpo_data/dpo-sa1b_0000_validated_vqa_gen-gemini_judge-gemini.jsonl \
    --seq_kd false \
    --lmbda 1 \
    --beta 1 \
    --torch_dtype bfloat16 \
    --num_train_epochs 1 \
    --per_device_train_batch_size 1 \
    --learning_rate 1e-5 \
    --gradient_accumulation_steps 1 \
    --save_steps 1000 \
    --save_total_limit 2 \
    --logging_steps 1 \
    --max_length 16000 \
    --max_completion_length 8192 \
    --output_dir output \
    --warmup_ratio 0.05 \
    --save_only_model true \
    --dataloader_num_workers 64 \
    --dataset_num_proc 4 \
    --deepspeed zero2 \
    --teacher_deepspeed zero3 \
    --attn_impl flash_attn \
    --use_vllm true \
    --vllm_mode server \
    --vllm_server_host 0.0.0.0 \
    --vllm_server_port 18924 \
    --output_dir /mnt/nas4/yanlong/code/swift-ckpt \

训练的loss感觉不太正常：
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 151645, 'bos_token_id': None}.
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 151645, 'bos_token_id': None}.
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 151645, 'bos_token_id': None}.
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 151645, 'bos_token_id': None}.
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 151645, 'bos_token_id': None}.
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 151645, 'bos_token_id': None}.
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 151645, 'bos_token_id': None}.
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 151645, 'bos_token_id': None}.
Train: 0%| | 0/700 [00:00<?, ?it/s][INFO:swift] use_logits_to_keep: False
Could not estimate the number of tokens of the input, floating-point operations will not be computed
Could not estimate the number of tokens of the input, floating-point operations will not be computed
Could not estimate the number of tokens of the input, floating-point operations will not be computed
Could not estimate the number of tokens of the input, floating-point operations will not be computed
Could not estimate the number of tokens of the input, floating-point operations will not be computed
Could not estimate the number of tokens of the input, floating-point operations will not be computed
Could not estimate the number of tokens of the input, floating-point operations will not be computed
Could not estimate the number of tokens of the input, floating-point operations will not be computed
{'loss': 0.67138672, 'grad_norm': 10.44564915, 'learning_rate': 2.9e-07, 'epoch': 0.0, 'global_step/max_steps': '1/700', 'elapsed_time': '2m 19s', 'remaining_time': '1d 2h 59m 17s', 'memory(GiB)': 57.27, 'train_speed(s/it)': 138.994834}
{'loss': 9.9765625, 'grad_norm': 114.45767212, 'learning_rate': 5.7e-07, 'epoch': 0.0, 'global_step/max_steps': '2/700', 'elapsed_time': '4m 27s', 'remaining_time': '1d 1h 54m 52s', 'memory(GiB)': 65.33, 'train_speed(s/it)': 133.656429}
{'loss': 10.203125, 'grad_norm': 118.75759888, 'learning_rate': 8.6e-07, 'epoch': 0.0, 'global_step/max_steps': '3/700', 'elapsed_time': '6m 37s', 'remaining_time': '1d 1h 37m 53s', 'memory(GiB)': 65.33, 'train_speed(s/it)': 132.385865}
{'loss': 9.953125, 'grad_norm': 107.8883667, 'learning_rate': 1.14e-06, 'epoch': 0.01, 'global_step/max_steps': '4/700', 'elapsed_time': '8m 45s', 'remaining_time': '1d 1h 22m 43s', 'memory(GiB)': 65.33, 'train_speed(s/it)': 131.26914}
{'loss': 9.9296875, 'grad_norm': 117.14516449, 'learning_rate': 1.43e-06, 'epoch': 0.01, 'global_step/max_steps': '5/700', 'elapsed_time': '10m 53s', 'remaining_time': '1d 1h 12m 53s', 'memory(GiB)': 65.33, 'train_speed(s/it)': 130.607949}
{'loss': 9.875, 'grad_norm': 114.77312469, 'learning_rate': 1.71e-06, 'epoch': 0.01, 'global_step/max_steps': '6/700', 'elapsed_time': '13m 3s', 'remaining_time': '1d 1h 9m 32s', 'memory(GiB)': 65.33, 'train_speed(s/it)': 130.507576}
{'loss': 9.578125, 'grad_norm': 137.29380798, 'learning_rate': 2e-06, 'epoch': 0.01, 'global_step/max_steps': '7/700', 'elapsed_time': '15m 17s', 'remaining_time': '1d 1h 12m 41s', 'memory(GiB)': 65.33, 'train_speed(s/it)': 130.96823}
{'loss': 9.375, 'grad_norm': 134.3533783, 'learning_rate': 2.29e-06, 'epoch': 0.01, 'global_step/max_steps': '8/700', 'elapsed_time': '17m 25s', 'remaining_time': '1d 1h 6m 48s', 'memory(GiB)': 65.33, 'train_speed(s/it)': 130.648101}
{'loss': 9.375, 'grad_norm': 135.55368042, 'learning_rate': 2.57e-06, 'epoch': 0.01, 'global_step/max_steps': '9/700', 'elapsed_time': '19m 33s', 'remaining_time': '1d 1h 0m 34s', 'memory(GiB)': 65.33, 'train_speed(s/it)': 130.295705}
Train: 1%|███▏ | 9/700 [19:32<24:48:49, 129.28s/it]

请问是什么原因呢

No source files or tests are named. Start by reproducing the Qwen3-VL GKD configuration and inspect the tokenizer/model token warnings, vLLM server setup, and early loss logs. Done means identifying whether the warnings or training configuration explain the abnormal loss and documenting a verified correction or cause.