Qwen3-VL GRPO: shape mismatch when using a large gradient accumulation step

Environments:

ms-swift==3b057eca3c54295feff75924be0cef8a7ac07bc6
transformers==4.57.1
vllm_ascend==0.11.0rc1.dev176+g38afd2c9c

I use the main branch, and do GRPO under server mode with 7 cards for training and 1 card for inference.

The rollout command is

swift rollout --model /cache/Qwen3-VL-8B-Instruct/ --vllm_enable_prefix_caching false --vllm_max_model_len 4096 --vllm_gpu_memory_utilization 0.8 --vllm_max_num_seqs 128

The script is

export IMAGE_MAX_TOKEN_NUM=1024
export USE_OPTIMIZED_MODEL=0
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6
export NPROC_PER_NODE=7

swift rlhf \
  --rlhf_type grpo \
  --model /cache/Qwen3-VL-8B-Instruct/ \
  --gradient_checkpointing_kwargs '{"use_reentrant": false}' \
  --train_type lora \
  --use_vllm true \
  --vllm_mode server \
  --vllm_server_host 127.0.0.1 \
  --vllm_server_port 8000 \
  --dataset lmms-lab/multimodal-open-r1-8k-verified#1000 \
  --load_from_cache_file true \
  --external_plugins examples/train/grpo/plugin/plugin.py \
  --reward_funcs external_r1v_acc format \
  --reward_weights 1 0.1 \
  --torch_dtype bfloat16 \
  --attn_impl flash_attn \
  --num_train_epochs 1 \
  --lora_rank 128 \
  --lora_alpha 256 \
  --max_length 4096 \
  --per_device_train_batch_size 1 \
  --per_device_eval_batch_size 1 \
  --gradient_accumulation_steps 32 \
  --eval_steps 500 \
  --save_steps 500 \
  --learning_rate 5e-6 \
  --save_total_limit 2 \
  --logging_steps 1 \
  --warmup_ratio 0.0 \
  --dataloader_num_workers 4 \
  --max_completion_length 512 \
  --num_generations 16 \
  --steps_per_generation 32 \
  --deepspeed zero2 \
  --temperature 1.1 \
  --top_p 1.0 \
  --top_k 80 \
  --log_completions false \
  --async_generate false \
  --system examples/train/grpo/prompt.txt \
  --beta 0.001 

The error is

[rank1]: Traceback (most recent call last):
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/cli/rlhf.py", line 5, in <module>
[rank1]:     rlhf_main()
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/llm/train/rlhf.py", line 227, in rlhf_main
[rank1]:     return SwiftRLHF(args).main()
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/llm/base.py", line 49, in main
[rank1]:     result = self.run()
[rank1]:              ^^^^^^^^^^
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/ray/base.py", line 170, in wrapper
[rank1]:     return func(self, *args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/llm/train/sft.py", line 206, in run
[rank1]:     return self.train(trainer)
[rank1]:            ^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/llm/train/sft.py", line 254, in train
[rank1]:     trainer.train(trainer.args.resume_from_checkpoint)
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/trainers/mixin.py", line 815, in train
[rank1]:     res = super().train(*args, **kwargs)
[rank1]:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/usr/local/python3.11.13/lib/python3.11/site-packages/transformers/trainer.py", line 2325, in train
[rank1]:     return inner_training_loop(
[rank1]:            ^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/usr/local/python3.11.13/lib/python3.11/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
[rank1]:     tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank1]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 1502, in training_step
[rank1]:     return super().training_step(model, inputs, num_items_in_batch)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/usr/local/python3.11.13/lib/python3.11/site-packages/transformers/trainer.py", line 4014, in training_step
[rank1]:     inputs = self._prepare_inputs(inputs)
[rank1]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 407, in wrapper
[rank1]:     return func(self, *args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 174, in _prepare_inputs
[rank1]:     generation_batch = self._generate_and_score_completions(generation_batch)
[rank1]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 407, in wrapper
[rank1]:     return func(self, *args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 225, in _generate_and_score_completions
[rank1]:     batch_encoded_inputs = self._prepare_batch_inputs(inputs)
[rank1]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 407, in wrapper
[rank1]:     return func(self, *args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 825, in _prepare_batch_inputs
[rank1]:     self._get_per_token_logps_and_entropies(self.model, batch_encoded_inputs)[0]
[rank1]:     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 407, in wrapper
[rank1]:     return func(self, *args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 1309, in _get_per_token_logps_and_entropies
[rank1]:     return self._get_per_token_logps_and_entropies_single(model, inputs, compute_entropy=compute_entropy)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 1351, in _get_per_token_logps_and_entropies_single
[rank1]:     logits = model(**inputs).logits
[rank1]:              ^^^^^^^^^^^^^^^
[rank1]:   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank1]:     return self._call_impl(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1857, in _call_impl
[rank1]:     return inner()
[rank1]:            ^^^^^^^
[rank1]:   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1805, in inner
[rank1]:     result = forward_call(*args, **kwargs)
[rank1]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/ma-user/.local/lib/python3.11/site-packages/peft/peft_model.py", line 1850, in forward
[rank1]:     return self.base_model(
[rank1]:            ^^^^^^^^^^^^^^^^
[rank1]:   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank1]:     return self._call_impl(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank1]:     return forward_call(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/ma-user/.local/lib/python3.11/site-packages/peft/tuners/tuners_utils.py", line 222, in forward
[rank1]:     return self.model.forward(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/usr/local/python3.11.13/lib/python3.11/site-packages/transformers/utils/generic.py", line 1064, in wrapper
[rank1]:     outputs = func(self, *args, **kwargs)
[rank1]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/usr/local/python3.11.13/lib/python3.11/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py", line 1344, in forward
[rank1]:     outputs = self.model(
[rank1]:               ^^^^^^^^^^^
[rank1]:   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank1]:     return self._call_impl(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank1]:     return forward_call(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/usr/local/python3.11.13/lib/python3.11/site-packages/transformers/utils/generic.py", line 1064, in wrapper
[rank1]:     outputs = func(self, *args, **kwargs)
[rank1]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/llm/model/model/qwen.py", line 1036, in forward
[rank1]:     outputs = self.language_model(
[rank1]:               ^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank1]:     return self._call_impl(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank1]:     return forward_call(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/usr/local/python3.11.13/lib/python3.11/site-packages/transformers/utils/generic.py", line 1064, in wrapper
[rank1]:     outputs = func(self, *args, **kwargs)
[rank1]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/usr/local/python3.11.13/lib/python3.11/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py", line 863, in forward
[rank1]:     hidden_states = self._deepstack_process(
[rank1]:                     ^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/ma-user/work/ms-swift-main-1105/ms-swift/swift/llm/model/model/qwen.py", line 945, in _deepstack_process
[rank1]:     local_this = hidden_states[visual_pos_masks, :].clone() + visual_embeds
[rank1]:                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~
[rank1]: RuntimeError: The size of tensor a (301) must match the size of tensor b (300) at non-singleton dimension 0
[ERROR] 2025-11-10-15:45:56 (PID:3903281, Device:1, RankID:-1) ERR99999 UNKNOWN applicaiton exception

When gradient_accumulation_steps and steps_per_generation are set to 16, the error is

[rank5]: RuntimeError: The size of tensor a (987) must match the size of tensor b (986) at non-singleton dimension 0