[discussions]: [Compatibility/vLLM] AutoScheme mixed-bit export (llm_compressor) fails in vLLM with AssertionError in load_merged_column_weight
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 175
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 99
Description
### Problem Description
### Describe the Issue
I am using `AutoScheme` to perform mixed-bit quantization targeting an average of 3.5 bits on `Qwen/Qwen3.5-4B` and exporting via `format="llm_compressor"`.
The quantization and export finish successfully, but when serving the exported model with **vLLM (v0.27.1)**, vLLM crashes during the weight-loading phase with an `AssertionError` in `load_merged_column_weight`:
```text
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/parameter.py", line 175, in load_merged_column_weight
assert param_data.shape == loaded_weight.shape
AssertionError
```
### Quantization Script
```python
import os
import torch
from auto_round import AutoRound, AutoScheme
from transformers import AutoModelForImageTextToText, AutoProcessor
MODEL_ID = "Qwen/Qwen3.5-4B"
OUTPUT_BASE_DIR = "./AutoRound"
LOCAL_PATH = "./local_model"
model = AutoModelForImageTextToText.from_pretrained(
LOCAL_PATH,
dtype=torch.bfloat16,
device_map="auto"
)
processor = AutoProcessor.from_pretrained(LOCAL_PATH)
tokenizer = processor.tokenizer
TUNING_CONFIG = {
"group_size": 64,
"sym": True,
"iters": 600,
"nsamples": 512,
"batch_size": 8,
"seqlen": 2048,
"low_gpu_mem_usage": False,
"enable_torch_compile": True,
"quant_nontext_module": False,
"layer_config": {
"mtp": {"data_type": "bfloat16"},
"mtp.fc": {"data_type": "bfloat16"}
}
}
mixed_scheme = AutoScheme(
avg_bits=3.5,
options=["W4A16", "W3A16", "W2A16"]
)
ar = AutoRound(
model=model,
tokenizer=tokenizer,
processor=processor,
scheme=mixed_scheme,
**TUNING_CONFIG,
)
ar.quantize_and_save(
OUTPUT_BASE_DIR,
format="llm_compressor",
inplace=True
)
```
### Questions / Guidance Needed
1. **Mixed-Bit Alignment for Fused Projections:** Does AutoScheme need `shared_layers` explicitly configured to prevent fusing projections (e.g., `in_proj_qkv`, `gate_proj`, `up_proj`) with mismatched bit-widths?
2. **Linear Attention & Recurrent Gates:** Should linear attention state gates (`in_proj_a`, `in_proj_b`) and `lm_head` be excluded/ignored from quantization when targeting vLLM's `compressed-tensors` backend?
3. **Supported Options:** Is `W2A16` or `group_size=64` supported by vLLM's `compressed-tensors` loader for Qwen hybrid models, or should `options` be restricted to `["W4A16", "W3A16"]` with `group_size=128`?
### Reproduction Steps
```bash
vllm serve Vishva007/Qwen3.5-4B-Mixed-3.5bit-AutoRound \
--kv-cache-dtype fp8 \
--max-model-len 32000 \
--trust-remote-code
```
### Environment Information
* **AutoRound Version:** Latest (`git+https://github.com/intel/auto-round.git`)
* **vLLM Version:** 0.27.1
* **Transformers Version:** 5.x / 4.x
* **PyTorch Version:** 2.x
* **GPU:** NVIDIA (48GB VRAM)
### Error Logs
```shell
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/default_loader.py", line 427, in load_weights
loaded_weights = model.load_weights(self.get_all_weights(model_config, model))
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_5.py", line 571, in load_weights
return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py", line 989, in weight_loader_v2
param.load_merged_column_weight(param, loaded_weight, shard_id)
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/parameter.py", line 175, in load_merged_column_weight
assert param_data.shape == loaded_weight.shape
AssertionError
```
### Additional Context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.