intel / intel/llm-scaler

[Bug] torch.compile fails on FP8 + XPU: `torch_ipex.fp8_gemm_w8a16` lacks fake-tensor implementation, forcing --enforce-eager (~10–20% perf cost)

Open
#383 2 comments 0 reactions 1 assignee Claimed by @xiangyuT View on GitHub
Dominant language
C++
Stars
529
Forks
80
Avg merge
9h 7m
Merged PRs (30d)
38

Description

## Summary

vLLM's memory profiling step (`determine_available_memory` → dummy forward pass through `torch.compile`) fails when running any FP8-quantized model on Intel XPU, because `torch.ops.torch_ipex.fp8_gemm_w8a16` does not register a fake-tensor (`meta`) implementation. The error surfaces from torch dynamo as:

```
torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors
Developer debug context: unsupported operator: torch_ipex.fp8_gemm_w8a16.xpu
```

This forces every FP8 deployment on XPU to launch with `--enforce-eager`, which disables `torch.compile` and CUDA-graph capture entirely and costs roughly 10–20% inference throughput. Outputs are correct in eager mode, but the user pays a quiet performance tax that most won't realize they're paying.

## Environment

| Item | Value |
|---|---|
| Image | `intel/llm-scaler-vllm:0.10.2-b6` |
| Internal vLLM | `0.10.3.dev0+g01efc7ef7.d20251125.xpu` |
| Hardware | 2× Intel Arc Pro B70 (Battlemage G31, `[8086:e223]`), PCIe (no XeLink) |
| Host OS | Ubuntu 26.04 LTS, kernel `7.0.0-14-generic` |
| Tensor parallel | TP=2 |
| Model exhibiting bug | `Qwen/Qwen3-VL-30B-A3B-Instruct` (MoE, 30B/3B-active) |
| Quantization | `--quantization fp8` (online dynamic) |

## Steps to reproduce

```bash
docker run --rm --name lsv \
--privileged --net=host --device=/dev/dri --shm-size=32g \
-v $HOME/.cache/huggingface:/root/.cache/huggingface \
-e HF_HOME=/root/.cache/huggingface \
--entrypoint /bin/bash \
intel/llm-scaler-vllm:0.10.2-b6 \
-lc 'vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
--tensor-parallel-size 2 \
--quantization fp8 \
--max-model-len 32768 \
--gpu-memory-utilization 0.85 \
--trust-remote-code'
```

Weights successfully download (~60 GB BF16) and load into VRAM (~20 GB per card under TP=2). The crash occurs during the memory-profiling dummy forward pass, before `/v1/models` becomes available. Under `systemd`-managed deployment with `Restart=on-failure` the failure is deterministic and produces a slow crash-restart loop (each cycle ~5–10 minutes depending on whether weights load from cache or re-download).

## Full error trace (verbatim from journald)

```
(Worker_TP0 pid=224) ERROR ... [multiproc_executor.py:654]
File ".../vllm/model_executor/models/qwen3_vl_moe.py", line 104, in forward
hidden_states, residual = layer(...)
File ".../vllm/model_executor/models/qwen3_moe.py", line 368, in forward
hidden_states = self.self_attn(...)
File ".../vllm/model_executor/models/qwen3_moe.py", line 278, in forward
qkv, _ = self.qkv_proj(hidden_states)
File ".../vllm/model_executor/layers/linear.py", line 520, in forward
output_parallel = self.quant_method.apply(self, input_, bias)
File ".../vllm/model_executor/layers/quantization/ipex_quant.py", line 307, in apply
output = torch.ops.torch_ipex.fp8_gemm_w8a16(x, weight, True, ...)

torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors
Developer debug context: unsupported operator: torch_ipex.fp8_gemm_w8a16.xpu
```

This propagates to the engine-core layer:
```
RuntimeError: Worker failed with error 'Operator does not support running with fake tensors'
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
```

## Workaround (and its cost)

Adding `--enforce-eager` to the launch flags allows the serve to come up successfully:

```
[t=30s] /v1/models=000 hf=58G gpu0=3.4GB gpu1=2.3GB
[t=60s] /v1/models=000 hf=58G gpu0=20.3GB gpu1=17.5GB ← weights loaded, TP=2 sharded
[t=90s] /v1/models=200 hf=58G gpu0=27.4GB gpu1=26.3GB ← serving
```

Both priority=100 (user-tier) and priority=0 (background-tier) chat completions return correct, well-formed text. TP=2 sharding works correctly under load. The model is fully functional in eager mode.

The cost is silent: no `torch.compile` graph fusion and no CUDA-graph capture. Practically, this is roughly 10–20% lower decode throughput and slightly higher TTFT compared to the compile-mode performance the hardware is otherwise capable of delivering.

## Impact

This affects every FP8-quantized model on XPU served through this image and likely earlier/later versions on the same vLLM line. The workaround is well-known among users who hit it, but it isn't documented as the *required* default for FP8+XPU, so users repeatedly discover it only after their service crash-loops in production.

## Suggested fix direction

Register a fake-tensor (`meta`) implementation for the operator. The standard PyTorch pattern:

```python
import torch

@torch.library.impl_abstract("torch_ipex::fp8_gemm_w8a16")
def _fp8_gemm_w8a16_fake(x, weight, has_bias, *args, **kwargs):
# Output shape: (*x.shape[:-1], weight.shape[0]); dtype matches the real op's output
return torch.empty(
(*x.shape[:-1], weight.shape[0]),
dtype=x.dtype,
device=x.device,
)
```

The same pattern would apply to any sibling IPEX XPU operators that surface during `torch.compile` tracing without `meta` registrations. Once registered, the operator is traceable through dynamo's fake-tensor pass without changing actual GEMM behavior, allowing `torch.compile` to proceed and unlocking the compile-mode performance for FP8 + XPU users.

## Related

- [vllm-project/vllm#38884](https://github.com/vllm-project/vllm/issues/38884) — same class of `torch._dynamo.exc.Unsupported … fake tensors` bug, different model (Gemma 4), different missing operator. Same documented workaround (`enforce_eager=True`).
- [intel/llm-scaler#382](https://github.com/intel/llm-scaler/issues/382) — same hardware (2× B70), same image, same FP8+TP=2 setup. User already has `--enforce-eager` set and hits a downstream `UR_RESULT_ERROR_OUT_OF_RESOURCES` during vision-encoder memory profiling. Suggests further fake-tensor or memory-profiling issues exist beyond `fp8_gemm_w8a16`.

Happy to test a candidate fix on the same hardware if useful.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.