huggingface / huggingface/trl

[GRPO] AssertionError in vLLM when using QLoRA (4-bit) due to weight shape mismatch during merge_adapter

Open
#4,973 3 comments 0 reactions 0 assignees View on GitHub
⚡ vLLM
Dominant language
Python
Stars
19.3k
Forks
3k
Avg merge
1d 20h
Merged PRs (30d)
194

Description

### Reproduction

I am trying to train a model using GRPOTrainer with QLoRA (4-bit quantization) and vLLM enabled (use_vllm=True). However, the training fails during the generation step when TRL attempts to synchronize weights with vLLM.

GRPOTrainer calls self.model.merge_adapter() to merge LoRA weights before generation.

Because the base model is loaded in 4-bit (BitsAndBytesConfig), merging the LoRA adapter forces the model to dequantize (upcast) the weights to bf16/fp16 to perform the addition.

TRL then sends these dequantized (unpacked) weights to vLLM via self.vllm_client.update_named_param.

However, the vLLM engine was initialized with 4-bit quantization (to match the training setup). It expects packed 4-bit weights (e.g., shape [out_features, in_features // 2]), but receives the dequantized full-shape weights (e.g., [out_features, in_features]).

This triggers an AssertionError in vLLM's weight_loader.

My commands:

for the server:
```
CUDA_VISIBLE_DEVICES=0 trl vllm-serve \
--model llm/models/Qwen2.5-3B-Instruct
```

for the training:
```
accelerate launch trl_test.py
```

My error:
```
File "/.../trl/trainer/grpo_trainer.py", line 1306, in _generate_single_turn
self._move_model_to_vllm()
File "/.../trl/trainer/grpo_trainer.py", line 1137, in _move_model_to_vllm
llm_model.load_weights([(name, param.data)])
...
File "/.../vllm/model_executor/layers/linear.py", line 1237, in weight_loader
assert param_data.shape == loaded_weight.shape
AssertionError
```

### System Info

- Python version: 3.10.19
- TRL version: 0.27.1
- PyTorch version: 2.9.0
- accelerator(s): NVIDIA GeForce RTX 4090, NVIDIA GeForce RTX 4090
- Transformers version: 4.57.6
- Accelerate version: 1.12.0
- Accelerate config: not found
- Datasets version: 4.5.0
- HF Hub version: 0.36.1
- bitsandbytes version: 0.49.1
- DeepSpeed version: 0.18.5
- Liger-Kernel version: not installed
- LLM-Blender version: not installed
- OpenAI version: 1.109.1
- PEFT version: 0.18.1
- vLLM version: 0.12.0

### Checklist

- [x] I have checked that my issue isn't already filed (see [open issues](https://github.com/huggingface/trl/issues?q=is%3Aissue))
- [x] I have included my system information
- [x] Any code provided is minimal, complete, and reproducible ([more on MREs](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks))
- [x] Any code provided is properly formatted in code blocks, (no screenshot, [more on code blocks](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks))
- [x] Any traceback provided is complete

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.