intel / intel/llm-scaler

AWQ quantization models have abnormal memory usage / OOM on PTL 358H

Open
#402 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
529
Forks
80
Avg merge
9h 7m
Merged PRs (30d)
38

Description

## Summary

When loading Qwen3.5 AWQ offline-quantized models with the `llm-scaler-vllm` image on PTL 358H, the memory footprint of the 9B model matches the FP8 baseline (no memory savings from AWQ), and the 35B-A3B model OOMs.

## Environment

- **Platform:** PTL 358H (16 cores); iGPU: B390 Graphics, 12 Xe Cores, 122 TOPS
- **Docker image:** `intel/llm-scaler-vllm:0.14.0-b7.1`
- **HF models:**
- `QuantTrio/Qwen3.5-9B-AWQ`
- `QuantTrio/Qwen3.5-35B-A3B-AWQ`

## Reproduction

Entrypoint script:

```bash
TORCH_LLM_ALLREDUCE=1 \
VLLM_USE_V1=1 \
CCL_ZE_IPC_EXCHANGE=pidfd \
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
python3 -m vllm.entrypoints.openai.api_server \
--model ${MODEL_PATH} \
--served-model-name ${SERVED_MODEL_NAME} \
--enforce-eager \
--port 8000 \
--host 0.0.0.0 \
--trust-remote-code \
--gpu-memory-util=0.6 \
--max-num-batched-tokens=8192 \
--disable-log-requests \
--max-model-len=${MAX_MODEL_LEN} \
--block-size 64 \
--quantization awq \
-tp=1 \
--enable_prefix_caching \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--allow-deprecated-quantization ipex_awq
```
## Observed behavior
QuantTrio/Qwen3.5-9B-AWQ — memory usage is roughly equivalent to the FP8 variant, i.e. AWQ appears to bring no memory reduction. See attached log: load_qwen3.5_9b_awq.log.
QuantTrio/Qwen3.5-35B-A3B-AWQ — OOM on load.
## Expected behavior
AWQ quantized weights should fit in significantly less memory than FP8, and the 35B-A3B AWQ model should load within the iGPU memory budget on this platform.

## Attachments

[load_qwen3.5_9b_awq.log](https://github.com/user-attachments/files/27584286/load_qwen3.5_9b_awq.log)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.