huggingface / huggingface/optimum-intel
Bug: Qwen3.5-9B INT4 export produces incoherent output (hybrid linear/full attention model)
- Dominant language
- Jupyter Notebook
- Stars
- 620
- Forks
- 270
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 22
Description
## Bug: Qwen3.5-9B INT4 export produces incoherent output (hybrid linear/full attention model)
### System Info
- optimum-intel: 1.27.0.dev0 (from git main, commit d4dd21a)
- openvino: 2026.2.0.dev20260501 (nightly)
- openvino-genai: 2026.2.0.0 (built from source, PR #3801 branch)
- transformers: 5.8.0.dev0 (git main)
- nncf: 2.16.0
- OS: Ubuntu 26.04
- Hardware: Intel Arc Pro B70 (24GB VRAM), 32 CPU cores, 30GB RAM
- compute-runtime: 26.14.37833.4
- IGC: 2.32.7
### Model
Qwen/Qwen3.5-9B — a hybrid linear-attention + full-attention VLM using Gated Delta Rule layers (32 layers in 3:1 linear:full ratio, `linear_conv_kernel_dim=4`, `mamba_ssm_dtype=float32`)
### Export command
```bash
optimum-cli export openvino -m Qwen/Qwen3.5-9B --weight-format int4 --task visual-language-modeling /data/models/qwen35-9b-int4-ov
```
### Description
After exporting Qwen3.5-9B with INT4 quantization, the model produces completely incoherent output on **all devices** (GPU and CPU), using **all inference paths** (openvino-genai ContinuousBatchingPipeline and optimum-intel OVModelForVisualCausalLM).
The same setup with the Qwen3.5-0.8B model exported as FP16 produces correct output ("What is 2+2?" → "4"), confirming that the runtime, drivers, and inference code are all functional.
### Reproduction
```python
import time
import torch
from optimum.intel import OVModelForVisualCausalLM
from transformers import AutoTokenizer
MODEL_DIR = "/data/models/qwen35-9b-int4-ov"
tokenizer = AutoTokenizer.from_pretrained(MODEL_DIR, trust_remote_code=True)
model = OVModelForVisualCausalLM.from_pretrained(MODEL_DIR, device="GPU", trust_remote_code=True)
messages = [{"role": "user", "content": "What is 2+2? Answer with just the number."}]
prompt_text = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, tokenize=False,
extra_context={"enable_thinking": False}
)
inputs = tokenizer(prompt_text, return_tensors="pt")
prompt_tokens = inputs["input_ids"].shape[-1]
# cache_position workaround for transformers 5.8 compatibility
cache_position = torch.arange(prompt_tokens)
output_ids = model.generate(**inputs, max_new_tokens=32, do_sample=False, cache_position=cache_position)
new_ids = output_ids[0][prompt_tokens:]
print(tokenizer.decode(new_ids, skip_special_tokens=True))
```
### Observed output
All three inference paths produce garbage:
| Inference Path | Device | Output |
|---|---|---|
| openvino_genai ContinuousBatchingPipeline | GPU | `"Thinking about 20. # 20. # 20000000000000..."` |
| optimum-intel OVModelForVisualCausalLM | GPU | `"ThinkingThinkingThinkingThinking..."` |
| optimum-intel OVModelForVisualCausalLM | CPU | `"\|,(,,, are,,,,,"` |
### Expected output
`4` (or similar correct response)
### Working comparison
The same code with Qwen3.5-0.8B (FP16, non-MoE) produces correct output:
```
Generated 4 tokens in 0.54s (7.4 tok/s)
Response: 4
```
### Analysis
Since the issue manifests on both CPU and GPU with two completely different inference engines, the problem is in the exported model weights — not the runtime. The INT4 quantization appears to corrupt the novel linear-attention / Gated Delta Rule layers. The 0.8B model works because it is smaller (same architecture but potentially fewer quantization-sensitive parameters at that scale).
Qwen3.5 uses a hybrid architecture with:
- `linear_attention` layers (Gated Delta Rule with conv kernel dim 4)
- `full_attention` layers (standard transformer attention)
- State-space-like components (`mamba_ssm_dtype: float32`)
The linear attention layers have different weight structures from standard transformers (`linear_key_head_dim`, `linear_value_head_dim`, `linear_num_key_heads`, `linear_num_value_heads`) which may not be handled correctly by the INT4 quantization pipeline.
### Questions
1. Have the Gated Delta Rule / linear attention weights been validated for INT4 quantization sensitivity?
2. Should certain layers (e.g., linear attention conv kernels, state projections) be excluded from quantization?
3. Would `--weight-format int8` or `--ratio 0.5` (mixed precision) avoid this issue?
4. Has the 0.8B model been tested with INT4 quantization (to confirm it's a scaling issue vs architectural)?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.