intel / intel/auto-round

[discussions]: [Compatibility/vLLM] AutoScheme mixed-bit export (llm_compressor) fails in vLLM with AssertionError in load_merged_column_weight

Open
#2,174 1 comment 0 reactions 1 assignee Claimed by @xin3he View on GitHub
bug
Dominant language
Python
Stars
1.6k
Forks
175
Avg merge
1d 18h
Merged PRs (30d)
99

Description

### Problem Description

### Describe the Issue

I am using `AutoScheme` to perform mixed-bit quantization targeting an average of 3.5 bits on `Qwen/Qwen3.5-4B` and exporting via `format="llm_compressor"`.

The quantization and export finish successfully, but when serving the exported model with **vLLM (v0.27.1)**, vLLM crashes during the weight-loading phase with an `AssertionError` in `load_merged_column_weight`:

```text
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/parameter.py", line 175, in load_merged_column_weight
assert param_data.shape == loaded_weight.shape
AssertionError

```

### Quantization Script

```python
import os
import torch
from auto_round import AutoRound, AutoScheme
from transformers import AutoModelForImageTextToText, AutoProcessor

MODEL_ID = "Qwen/Qwen3.5-4B"
OUTPUT_BASE_DIR = "./AutoRound"
LOCAL_PATH = "./local_model"

model = AutoModelForImageTextToText.from_pretrained(
LOCAL_PATH,
dtype=torch.bfloat16,
device_map="auto"
)
processor = AutoProcessor.from_pretrained(LOCAL_PATH)
tokenizer = processor.tokenizer

TUNING_CONFIG = {
"group_size": 64,
"sym": True,
"iters": 600,
"nsamples": 512,
"batch_size": 8,
"seqlen": 2048,
"low_gpu_mem_usage": False,
"enable_torch_compile": True,
"quant_nontext_module": False,
"layer_config": {
"mtp": {"data_type": "bfloat16"},
"mtp.fc": {"data_type": "bfloat16"}
}
}

mixed_scheme = AutoScheme(
avg_bits=3.5,
options=["W4A16", "W3A16", "W2A16"]
)

ar = AutoRound(
model=model,
tokenizer=tokenizer,
processor=processor,
scheme=mixed_scheme,
**TUNING_CONFIG,
)

ar.quantize_and_save(
OUTPUT_BASE_DIR,
format="llm_compressor",
inplace=True
)

```

### Questions / Guidance Needed

1. **Mixed-Bit Alignment for Fused Projections:** Does AutoScheme need `shared_layers` explicitly configured to prevent fusing projections (e.g., `in_proj_qkv`, `gate_proj`, `up_proj`) with mismatched bit-widths?
2. **Linear Attention & Recurrent Gates:** Should linear attention state gates (`in_proj_a`, `in_proj_b`) and `lm_head` be excluded/ignored from quantization when targeting vLLM's `compressed-tensors` backend?
3. **Supported Options:** Is `W2A16` or `group_size=64` supported by vLLM's `compressed-tensors` loader for Qwen hybrid models, or should `options` be restricted to `["W4A16", "W3A16"]` with `group_size=128`?

### Reproduction Steps

```bash
vllm serve Vishva007/Qwen3.5-4B-Mixed-3.5bit-AutoRound \
--kv-cache-dtype fp8 \
--max-model-len 32000 \
--trust-remote-code
```

### Environment Information

* **AutoRound Version:** Latest (`git+https://github.com/intel/auto-round.git`)
* **vLLM Version:** 0.27.1
* **Transformers Version:** 5.x / 4.x
* **PyTorch Version:** 2.x
* **GPU:** NVIDIA (48GB VRAM)

### Error Logs

```shell
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/default_loader.py", line 427, in load_weights
loaded_weights = model.load_weights(self.get_all_weights(model_config, model))
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_5.py", line 571, in load_weights
return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py", line 989, in weight_loader_v2
param.load_merged_column_weight(param, loaded_weight, shard_id)
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/parameter.py", line 175, in load_merged_column_weight
assert param_data.shape == loaded_weight.shape
AssertionError
```

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.