intel / intel/llm-scaler

0.26.0-b2: XPU Graph / torch.compile cannot start with sym_int4 — logger.info_once() inside the traced apply() path hard-fails Dynamo

Open
#699 2 comments 0 reactions 1 assignee Claimed by @gc-fu View on GitHub
Dominant language
C++
Stars
529
Forks
80
Avg merge
9h 7m
Merged PRs (30d)
38

Description

## Summary

On `intel/llm-scaler-vllm:0.26.0-b2`, **any compiled execution mode fails at engine
initialisation** when serving a `sym_int4` model. This is not a corruption issue — the server
never reaches graph capture:

```
torch._dynamo.exc.Unsupported: logging.Logger method not supported for non-export cases
Explanation: logging.Logger methods are not supported for non-export cases.
Hint: Add the logging method to `torch._dynamo.config.ignore_logging_functions`.
Developer debug context: method: .info_once
```

`vllm/model_executor/layers/quantization/sym_int4.py` calls `logger.info_once()` **inside the
traced `apply()` forward path** — two call sites in the `int4_gemm_w4a16` branch (around lines
396 and 402 in the shipped file):

```python
if output is not None:
logger.info_once(
"sym_int4 linear execution is using the guarded ESIMD path.", scope="local",
)
return output.reshape(...)

logger.info_once(
"sym_int4 linear execution is using the XPU W4A16 kernel.", scope="local",
)
output = torch.ops._xpu_C.int4_gemm_w4a16(...)
```

TorchDynamo refuses to trace a `logging.Logger` method and raises, aborting startup.

## Impact

This blocks **both** compiled modes, so it is broader than an XPU-Graph issue:
- `VLLM_XPU_ENABLE_XPU_GRAPH=1` (compile + graph) — fails
- compile-only (`cudagraph_mode=NONE`, XPU Graph off, no `--enforce-eager`) — fails

Only `--enforce-eager` starts. That makes it impossible to evaluate compiled performance or to
run a compile-vs-graph correctness comparison on this image without patching.

`TORCHDYNAMO_SUPPRESS_ERRORS=1` does **not** work around it: we verified
`torch._dynamo.config.suppress_errors == True` was live in the container, but the failing path
is `torch/_dynamo/aot_compile.py`, and AOT compilation raises rather than falling back to eager.

## Environment

| | |
|---|---|
| Image | `intel/llm-scaler-vllm:0.26.0-b2` (`sha256:52218ad85513ab6686d4c090c83c2bd8c5b02423c63aa4dabd41837fe641fe3b`) |
| vLLM | `0.26.1.dev0+g568afb3a1.d20260907` |
| torch | `2.12.0+xpu` |
| GPUs | 4x Intel Arc Pro B60 (Battlemage G21, `8086:e211`), 24 GB each |
| Model | `Qwen/Qwen3.6-35B-A3B`, revision `995ad96eacd98c81ed38be0c5b274b04031597b0` |
| Quant | online `sym_int4`, `--dtype float16` |
| TP | reproduces at `-tp 1`, `-tp 2` and `-tp 4` |
| Host | Ubuntu, kernel 7.0.0-31-generic, driver `xe`, Level Zero UMD 26.05.037020 |

## Reproduce

```bash
docker run -d --name repro --net=host --privileged \
--device /dev/dri:/dev/dri -v /dev/dri/by-path:/dev/dri/by-path \
-v /var/lib/models:/models:ro --shm-size 32g \
-e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 \
-e VLLM_WORKER_MULTIPROC_METHOD=spawn \
--entrypoint bash intel/llm-scaler-vllm:0.26.0-b2 \
-c "vllm serve /models/.../Qwen3.6-35B-A3B/snapshots/995ad96e... \
--dtype float16 --quantization sym_int4 --allow-deprecated-quantization \
--compilation-config '{\"cudagraph_capture_sizes\":[1,2,4,8,16]}' \
--max-model-len 8192 -tp 1 --max-num-seqs 32 --host 0.0.0.0 --port 8000"
```
Engine init aborts. Same with `--compilation-config '{"cudagraph_mode":"NONE"}'` and
`VLLM_XPU_ENABLE_XPU_GRAPH=0`.

## Suggested fix

Register the custom logger helpers with Dynamo where the module defines its logger — the
mechanism Dynamo's own hint names. Note the methods are attached to the logger **instance**
(`type(logger)` is plain `logging.Logger`), so they must be taken off the instance:

```python
logger = init_logger(__name__)

try:
import torch._dynamo.config as _dynamo_config
for _m in ("info_once", "warning_once", "debug_once", "error_once"):
_bound = getattr(logger, _m, None)
if _bound is None:
continue
_dynamo_config.ignore_logging_functions.add(_bound)
_raw = getattr(_bound, "__func__", None)
if _raw is not None:
_dynamo_config.ignore_logging_functions.add(_raw)
except Exception:
pass
```

It must be done **in-module**, not from the parent process: workers are spawned, so a mutation
made before `vllm serve` never reaches the process that actually compiles.

Alternatively, hoist the two `info_once()` calls out of `apply()` (e.g. log the selected kernel
once at layer construction), which avoids the traced-path call entirely.

## Verification of the workaround

With the registration above applied, on the same image and model:
- graph reaches capture (`Graph capturing finished in 9 secs, took 1.89 GiB`; PIECEWISE and FULL
decode graphs across capture sizes `[1,2,4,8,16]`)
- serves coherently at tp=1 (0/24 degenerate across two 12-run gates at 8k and 32k)
- **1.66-1.67x** output throughput vs a verified-eager control at identical context and
`max_num_seqs` (196.23 -> 328.01 tok/s at 8k; 198.10 -> 328.42 at 32k, c=8, in=512/out=256)

## Separate note (filed here only for context, not as part of this report)

Graph capture additionally requires `max_num_seqs` <= available mamba cache blocks:
```
ValueError: max_num_seqs (256) exceeds available Mamba cache blocks (44).
```
The default `max_num_seqs=256` therefore fails capture out of the box on this model at 24 GB.
Happy to file that separately if useful.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.