[Bug] torch.compile fails on FP8 + XPU: `torch_ipex.fp8_gemm_w8a16` lacks fake-tensor implementation, forcing --enforce-eager (~10–20% perf cost)
- Dominant language
- C++
- Stars
- 529
- Forks
- 80
- Avg merge
- 9h 7m
- Merged PRs (30d)
- 38
Description
## Summary
vLLM's memory profiling step (`determine_available_memory` → dummy forward pass through `torch.compile`) fails when running any FP8-quantized model on Intel XPU, because `torch.ops.torch_ipex.fp8_gemm_w8a16` does not register a fake-tensor (`meta`) implementation. The error surfaces from torch dynamo as:
```
torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors
Developer debug context: unsupported operator: torch_ipex.fp8_gemm_w8a16.xpu
```
This forces every FP8 deployment on XPU to launch with `--enforce-eager`, which disables `torch.compile` and CUDA-graph capture entirely and costs roughly 10–20% inference throughput. Outputs are correct in eager mode, but the user pays a quiet performance tax that most won't realize they're paying.
## Environment
| Item | Value |
|---|---|
| Image | `intel/llm-scaler-vllm:0.10.2-b6` |
| Internal vLLM | `0.10.3.dev0+g01efc7ef7.d20251125.xpu` |
| Hardware | 2× Intel Arc Pro B70 (Battlemage G31, `[8086:e223]`), PCIe (no XeLink) |
| Host OS | Ubuntu 26.04 LTS, kernel `7.0.0-14-generic` |
| Tensor parallel | TP=2 |
| Model exhibiting bug | `Qwen/Qwen3-VL-30B-A3B-Instruct` (MoE, 30B/3B-active) |
| Quantization | `--quantization fp8` (online dynamic) |
## Steps to reproduce
```bash
docker run --rm --name lsv \
--privileged --net=host --device=/dev/dri --shm-size=32g \
-v $HOME/.cache/huggingface:/root/.cache/huggingface \
-e HF_HOME=/root/.cache/huggingface \
--entrypoint /bin/bash \
intel/llm-scaler-vllm:0.10.2-b6 \
-lc 'vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
--tensor-parallel-size 2 \
--quantization fp8 \
--max-model-len 32768 \
--gpu-memory-utilization 0.85 \
--trust-remote-code'
```
Weights successfully download (~60 GB BF16) and load into VRAM (~20 GB per card under TP=2). The crash occurs during the memory-profiling dummy forward pass, before `/v1/models` becomes available. Under `systemd`-managed deployment with `Restart=on-failure` the failure is deterministic and produces a slow crash-restart loop (each cycle ~5–10 minutes depending on whether weights load from cache or re-download).
## Full error trace (verbatim from journald)
```
(Worker_TP0 pid=224) ERROR ... [multiproc_executor.py:654]
File ".../vllm/model_executor/models/qwen3_vl_moe.py", line 104, in forward
hidden_states, residual = layer(...)
File ".../vllm/model_executor/models/qwen3_moe.py", line 368, in forward
hidden_states = self.self_attn(...)
File ".../vllm/model_executor/models/qwen3_moe.py", line 278, in forward
qkv, _ = self.qkv_proj(hidden_states)
File ".../vllm/model_executor/layers/linear.py", line 520, in forward
output_parallel = self.quant_method.apply(self, input_, bias)
File ".../vllm/model_executor/layers/quantization/ipex_quant.py", line 307, in apply
output = torch.ops.torch_ipex.fp8_gemm_w8a16(x, weight, True, ...)
torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors
Developer debug context: unsupported operator: torch_ipex.fp8_gemm_w8a16.xpu
```
This propagates to the engine-core layer:
```
RuntimeError: Worker failed with error 'Operator does not support running with fake tensors'
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
```
## Workaround (and its cost)
Adding `--enforce-eager` to the launch flags allows the serve to come up successfully:
```
[t=30s] /v1/models=000 hf=58G gpu0=3.4GB gpu1=2.3GB
[t=60s] /v1/models=000 hf=58G gpu0=20.3GB gpu1=17.5GB ← weights loaded, TP=2 sharded
[t=90s] /v1/models=200 hf=58G gpu0=27.4GB gpu1=26.3GB ← serving
```
Both priority=100 (user-tier) and priority=0 (background-tier) chat completions return correct, well-formed text. TP=2 sharding works correctly under load. The model is fully functional in eager mode.
The cost is silent: no `torch.compile` graph fusion and no CUDA-graph capture. Practically, this is roughly 10–20% lower decode throughput and slightly higher TTFT compared to the compile-mode performance the hardware is otherwise capable of delivering.
## Impact
This affects every FP8-quantized model on XPU served through this image and likely earlier/later versions on the same vLLM line. The workaround is well-known among users who hit it, but it isn't documented as the *required* default for FP8+XPU, so users repeatedly discover it only after their service crash-loops in production.
## Suggested fix direction
Register a fake-tensor (`meta`) implementation for the operator. The standard PyTorch pattern:
```python
import torch
@torch.library.impl_abstract("torch_ipex::fp8_gemm_w8a16")
def _fp8_gemm_w8a16_fake(x, weight, has_bias, *args, **kwargs):
# Output shape: (*x.shape[:-1], weight.shape[0]); dtype matches the real op's output
return torch.empty(
(*x.shape[:-1], weight.shape[0]),
dtype=x.dtype,
device=x.device,
)
```
The same pattern would apply to any sibling IPEX XPU operators that surface during `torch.compile` tracing without `meta` registrations. Once registered, the operator is traceable through dynamo's fake-tensor pass without changing actual GEMM behavior, allowing `torch.compile` to proceed and unlocking the compile-mode performance for FP8 + XPU users.
## Related
- [vllm-project/vllm#38884](https://github.com/vllm-project/vllm/issues/38884) — same class of `torch._dynamo.exc.Unsupported … fake tensors` bug, different model (Gemma 4), different missing operator. Same documented workaround (`enforce_eager=True`).
- [intel/llm-scaler#382](https://github.com/intel/llm-scaler/issues/382) — same hardware (2× B70), same image, same FP8+TP=2 setup. User already has `--enforce-eager` set and hits a downstream `UR_RESULT_ERROR_OUT_OF_RESOURCES` during vision-encoder memory profiling. Suggests further fake-tensor or memory-profiling issues exist beyond `fp8_gemm_w8a16`.
Happy to test a candidate fix on the same hardware if useful.
Contributor guide
Assessment
This issue has not been assessed yet.