以seq_cls 微调qwen3_VL_8B时训练时间异常，比qwen25_VL_7B多10倍以上训练时间

Describe the bug
训练脚本
#!/bin/bash CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ NPROC_PER_NODE=8 \ VIDEO_MAX_TOKEN_NUM=256 \ FPS_MAX_FRAMES=32 \ FPS=10 \ swift sft \ --model Qwen/Qwen3-VL-8B-Instruct \ --train_type lora \ --dataset '/share/caibinquan-local/data/ms_dataset/Linkric20250306_dataset/train.json' \ --val_dataset '/share/caibinquan-local/data/ms_dataset/Linkric20250306_dataset/val.json' \ --load_from_cache_file true \ --torch_dtype bfloat16 \ --num_train_epochs 10 \ --per_device_train_batch_size 1 \ --per_device_eval_batch_size 1 \ --learning_rate 1e-4 \ --lora_rank 8 \ --lora_alpha 32 \ --target_modules all-linear \ --gradient_accumulation_steps 2 \ --eval_strategy epoch \ --save_strategy epoch \ --save_total_limit 2 \ --logging_steps 5 \ --max_length 5120 \ --output_dir 'output/qwen3_VL_seq-cls_sft' \ --warmup_ratio 0.05 \ --dataloader_num_workers 4 \ --num_labels 21 \ --task_type seq_cls \ --deepspeed zero2 \ --metric_for_best_model eval_acc

Your hardware and system info

主要库版本
decord 0.6.0
deepspeed 0.18.0
evalscope 1.1.0
evaluate 0.4.6
ms_swift 3.9.1 /share/caibinquan-nfs/ms-swift-3.9.1
numpy 1.26.4
nvidia-cublas-cu12 12.8.4.1
torch 2.9.0
torchvision 0.24.0
traitlets 5.14.3
transformers 4.57.1
trl 0.23.1

Additional context
qwen2.5_VL训练时间：
{"loss": 9.461e-05, "grad_norm": 0.00997003, "learning_rate": 0.0, "acc": 1.0, "epoch": 10.0, "global_step/max_steps": "3520/3520", "percentage": "100.00%", "elapsed_time": "5h 34m 3s", "remaining_time": "0s", "memory(GiB)": 21.26, "train_speed(iter/s)": 0.175616} {"eval_loss": 0.97969204, "eval_runtime": 695.1556, "eval_samples_per_second": 3.221, "eval_steps_per_second": 0.403, "eval_acc": 0.87633929, "epoch": 10.0, "global_step/max_steps": "3520/3520", "percentage": "100.00%", "elapsed_time": "5h 45m 38s", "remaining_time": "0s", "memory(GiB)": 21.26, "train_speed(iter/s)": 0.16973}

No source file or test is named. Start from the swift sft command in the report and compare the Qwen3-VL and Qwen2.5-VL seq_cls runs with the listed versions, eight-GPU configuration, and timing data. Done means the slowdown is reproducible and its cause or an actionable fix is documented with a confirming benchmark.