使用外部 vLLM 进行 GRPO 训练时，不能使用 QLoRA 吗？

Describe the bug
我想使用 external vLLM 和基于 bnb 的 QLoRA 来进行 GRPO 训练，并已经尝试开始训练，但在 rollout 服务器上出现了以下错误，导致训练无法继续。

(EngineCore_DP0 pid=492234)
(EngineCore_DP0 pid=492234) INFO 12-04 00:09:36 [punica_selector.py:19] Using PunicaWrapperGPU.
(EngineCore_DP0 pid=492234) WARNING 12-04 00:09:36 [models.py:62] For MoE models, vLLM currently does not support fused MoE LoRA inference. Please ensure that the loaded LoRA model does not contain expert weights.
(EngineCore_DP0 pid=492234) INFO 12-04 00:09:36 [gpu_model_runner.py:2653] Model loading took 16.9222 GiB and 873.428363 seconds
(EngineCore_DP0 pid=492234) INFO 12-04 00:11:20 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/452fff0fbd/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=492234) INFO 12-04 00:11:20 [backends.py:559] Dynamo bytecode transform time: 100.26 s
(EngineCore_DP0 pid=492234) INFO 12-04 00:11:29 [backends.py:197] Cache the graph for dynamic shape for later use
(EngineCore_DP0 pid=492234) INFO 12-04 00:12:22 [backends.py:218] Compiling a graph for dynamic shape takes 60.47 s
(EngineCore_DP0 pid=492234) WARNING 12-04 00:12:26 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=768,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition.json']
(EngineCore_DP0 pid=492234) INFO 12-04 00:12:27 [monitor.py:34] torch.compile takes 160.73 s in total
(EngineCore_DP0 pid=492234) INFO 12-04 00:12:29 [gpu_worker.py:298] Available KV cache memory: 67.08 GiB
(EngineCore_DP0 pid=492234) INFO 12-04 00:12:30 [kv_cache_utils.py:1087] GPU KV cache size: 732,640 tokens
(EngineCore_DP0 pid=492234) INFO 12-04 00:12:30 [kv_cache_utils.py:1091] Maximum concurrency for 262,144 tokens per request: 2.79x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 67/67 [00:21<00:00, 3.06it/s]
Capturing CUDA graphs (decode, FULL): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:09<00:00, 3.78it/s]
(EngineCore_DP0 pid=492234) INFO 12-04 00:13:02 [gpu_model_runner.py:3480] Graph capturing finished in 32 secs, took 2.16 GiB
(EngineCore_DP0 pid=492234) INFO 12-04 00:13:02 [core.py:210] init engine (profile, create kv cache, warmup model) took 205.57 seconds
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO: 127.0.0.1:50902 - "GET /health/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:50914 - "POST /close_communicator/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:50914 - "GET /get_world_size/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:50914 - "POST /init_communicator/ HTTP/1.1" 200 OK
[rank0]:[W1204 00:23:16.341809808 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3
(EngineCore_DP0 pid=492234) INFO 12-04 00:23:16 [init.py:1384] Found nccl from library libnccl.so.2
(EngineCore_DP0 pid=492234) INFO 12-04 00:23:16 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO: 127.0.0.1:58832 - "POST /get_engine_type/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:41072 - "POST /update_flattened_params/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:41072 - "POST /update_flattened_params/ HTTP/1.1" 200 OK
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] Invocation of collective_rpc method failed
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] Traceback (most recent call last):
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 777, in _handle_client_request
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] result = method(*self._convert_msgspec_args(method, args))
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 416, in collective_rpc
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] return self.model_executor.collective_rpc(method, timeout, args,
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/utils/init.py", line 3122, in run_method
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] return func(*args, **kwargs)
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/swift/llm/infer/rollout.py", line 133, in update_flattened_params
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] self.model_runner.model.load_weights(weights=list(named_params.items()))
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/model_executor/models/qwen3_moe.py", line 689, in load_weights
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] return loader.load_weights(weights)
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 294, in load_weights
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] autoloaded_weights = set(self._load_module("", self.module, weights))
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 252, in _load_module
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] yield from self._load_module(prefix,
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 225, in _load_module
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] loaded_params = module_load_weights(weights)
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/model_executor/models/qwen3_moe.py", line 563, in load_weights
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] param = params_dict[name]
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] ~~~~~~~~~~~^^^^^^
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] KeyError: 'layers.0.self_attn.qkqkv_proj.base_layer.weight.absmax'
[ERROR:swift] Method execution failed: collective_rpc
Traceback (most recent call last):
File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/swift/llm/infer/rollout.py", line 214, in llm_worker
result = method(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/v1/engine/llm_engine.py", line 362, in collective_rpc
return self.engine_core.collective_rpc(method, timeout, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 749, in collective_rpc
return self.call_utility("collective_rpc", method, timeout, args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 697, in call_utility
return future.result()
^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
Exception: Call to collective_rpc method failed: 'layers.0.self_attn.qkqkv_proj.base_layer.weight.absmax'

INFO: 127.0.0.1:41072 - "POST /update_flattened_params/ HTTP/1.1" 200 OK
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] Invocation of collective_rpc method failed
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] Traceback (most recent call last):
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 777, in _handle_client_request
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] result = method(*self._convert_msgspec_args(method, args))
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 416, in collective_rpc
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] return self.model_executor.collective_rpc(method, timeout, args,
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/utils/init.py", line 3122, in run_method
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] return func(*args, **kwargs)
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/swift/llm/infer/rollout.py", line 133, in update_flattened_params
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] self.model_runner.model.load_weights(weights=list(named_params.items()))
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/model_executor/models/qwen3_moe.py", line 689, in load_weights
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] return loader.load_weights(weights)
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 294, in load_weights
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] autoloaded_weights = set(self._load_module("", self.module, weights))
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 252, in _load_module
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] yield from self._load_module(prefix,
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 225, in _load_module
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] loaded_params = module_load_weights(weights)
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/model_executor/models/qwen3_moe.py", line 521, in load_weights
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] param = params_dict[name_mapped]
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] ~~~~~~~~~~~^^^^^^^^^^^^^
(EngineCore_DP0 pid=492234) ERROR 12-04 00:27:23 [core.py:780] KeyError: 'layers.1.mlp.experts.w13_base_layer.weight'
[ERROR:swift] Method execution failed: collective_rpc
Traceback (most recent call last):
File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/swift/llm/infer/rollout.py", line 214, in llm_worker
result = method(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/v1/engine/llm_engine.py", line 362, in collective_rpc
return self.engine_core.collective_rpc(method, timeout, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 749, in collective_rpc
return self.call_utility("collective_rpc", method, timeout, args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jovyan/workspace/chanwoo/omniverse-pipeline/venv/e2e-venv/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 697, in call_utility
return future.result()
^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
Exception: Call to collective_rpc method failed: 'layers.1.mlp.experts.w13_base_layer.weight'

Your hardware and system info
Write your system info like CUDA version/system/GPU/torch version here(在这里给出硬件信息和系统信息，如CUDA版本，系统，GPU型号和torch版本等)

CUDA version: 13.0
GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition (96gb)

torch 2.8.0
torchaudio 2.8.0
torchvision 0.23.0
transformers 4.57.0
transformers-stream-generator 0.0.5
ms_swift 3.10.3
vllm 0.11.0
bitsandbytes 0.48.2
trl 0.24.0

Additional context
用于 GRPO 训练和运行 rollout 服务器的参数值和指令如下：

rollout command:
CUDA_VISIBLE_DEVICES=7 swift rollout --model ./qwen3-30b-a3b-thinking-2507 --model_type qwen3_moe --max_new_tokens 8192 --quant_method bnb --quant_bits 4 --vllm_quantization bitsandbytes --vllm_enable_lora true

grpo config:
parallel_args:
nproc_per_node: 4
tensor_parallel_size: 1
training_args:
add_version: false
eval_steps: 1
save_steps: 1
model_type: qwen3_moe
num_train_epochs: 2
train_type: lora
torch_dtype: bfloat16
learning_rate: 1e-5
lora_rank: 16
lora_alpha: 64
quant_method: bnb
quant_bits: 4
bnb_4bit_compute_dtype: bfloat16
bnb_4bit_quant_type: nf4
bnb_4bit_use_double_quant: true
target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
- gate_proj
- up_proj
- down_proj
val_dataset: ./valid_data_only_planning.json
num_generations: 2
max_length: 8192
max_completion_length: 4096
per_device_train_batch_size: 2
per_device_eval_batch_size: 2
use_vllm: True
vllm_mode: server
vllm_server_host: 127.0.0.1
vllm_server_port: 8000
offload_model: True
offload_optimizer: True
external_plugins: ./reward_plugins.py
reward_funcs:
- reward_model
logging_steps: 1
log_completions: True
beta: 0.04
attn_impl: flash_attn

*我想训练的模型是 qwen3-30b-a3b-thinking-2507！
*出现上述错误的时间点，是在运行训练脚本的终端出现以下日志之后。
UserWarning: Merge lora module to 4-bit linear may get different generations due to rounding errors.
UserWarning: Unmerge lora module to 4-bit linear may get different generations due to rounding errors.

Start with swift/llm/infer/rollout.py at update_flattened_params, then inspect vLLM's qwen3_moe.py load_weights path shown in the traceback. Reproduce GRPO with external vLLM and bitsandbytes QLoRA, and determine whether the missing qkqkv_proj.base_layer.weight.absmax is an unsupported combination or a loading mismatch. Done means documenting the supported configuration or identifying the required compatibility change.